challenges in implementing case study

Via Giacomo Saudino, 36

10015 Ivrea, Italy

[email protected]

(+39) 334 855 0474.   |    [email protected]

The Implementation: Another Crucial, Ingredient for Case Study Success

by John Cole | Jun 21, 2021 | B2B Copywriting , Technology Marketing , Collateral , Lead Generation , Case Studies | 0 comments

When prospects talk with your salespeople about purchasing your solutions, what are some of their biggest concerns?

If you’re like most tech industry executives, I’ll bet your list includes:

• What will it take to implement your solution?
• How long will it take to get it up and running?
• What problems might we run into during the implementation?

Let’s face it. Unless your solution is extremely simple, there’s always an implementation phase. Tech buyers know that until they’ve completed that phase, there will be no ROI. And if the implementation doesn’t go well, there will be schedule delays, unplanned expenses, and lower returns.

So, your prospects want reassurance that implementation will be accomplished quickly and efficiently.

How can you help your sales and technical support staff address those concerns?

Well, if you publish case studies, you have the perfect vehicle. Once you’ve introduced your solution in the story, all you have to do is describe the implementation and how your company supported it. You’ll be supplying your sales reps with solid evidence—from real customers—that you’ve dealt with implementation problems successfully.

Strangely, many case studies fail to sufficiently cover this critical phase. We’ll examine why in a moment. Then, we’ll talk about how to make the most of your implementation discussion. But first, let’s look at why you should address the implementation process in your case studies.

Why the implementation matters in a case study

Like the customer Journey we discussed last time , the Implementation is important to your case study in two ways: first as a marketing force, and second as a structural element for creating drama—for making your customer’s story compelling.

From the marketing viewpoint, consider your audience. Tech buyers are a skeptical bunch. They know that no new system implementation comes off without a hitch. They expect unexpected obstacles. Any marketing claims of “easy installation and set-up” they take with a big grain of salt.

So, in your case studies, it’s important to describe the implementation in detail.

You need to talk about any difficulties or obstacles that were encountered and how they were overcome. This builds credibility with tech buyers. It answers some of their most important questions. It shows them you can handle those challenges. And it makes the results you describe later more believable.

It may seem counter-intuitive, but revealing the challenges in the implementation process—and how they were successfully resolved, of course—makes both your solution and your story more compelling for your reader.

The storytelling importance of the implementation

Which leads us to the storytelling importance of the implementation…

One of the most important forces in any story is the Resolution of Conflict, which begins when the hero of the story—just when all seems lost—determines exactly what she must do. In a case study, that’s where the customer discovers your solution. The implementation, then, is the next logical step. It shows our hero putting her plan into action. The implementation drives the story to its climax.

If you jump too quickly from purchase to results without spending enough time on the implementation, your story will feel hollow. Empty. Your results will seem too good to be true. Your readers need that resolution of the conflict so that they can believe in your “happily ever after.”

Why the implementation gets short-changed

So why does the implementation phase so often get short shrift in case studies? I think there are three reasons.

The most likely reason is that the Implementation, like the Journey, is not one of the four major divisions—i.e., the Customer, the Challenge, the Solution, and the Results—that we usually think of as the basic case study structure. Seasoned case study writers will usually discuss the Implementation as part of the Solution section. But other copywriters—like those from ad agencies, for example, who are trained to emphasize benefits above all else—may focus solely on the solution itself and not appreciate the importance of the Implementation.

A second possible reason is customer apprehension. Some customers may want to suppress or limit the implementation discussion because they fear they may lose a competitive advantage they’ve just gained. Others may feel they may somehow tarnish their corporate image if they reveal they needed help to implement your solution.

A third reason may lie with the solution provider—the company publishing the case study. If problems were encountered during implementation, they may fear it will reflect badly on their solution or their customer, They may, therefore, decide to suppress the implementation discussion.

Obstacles are Opportunities

But if you look at it from your prospect’s perspective, implementation challenges should be embraced in a case study.

Prospects want to know what they’ll be up against. They want to know what it took to successfully apply your solution to the customer’s application. They also want to know how your company helped the customer overcome any problems they encountered, and what the customer thought of the help they received. In other words, implementation problems are opportunities. You can easily turn them into credibility builders!

And when you think about it, problems encountered during implementation don’t really reflect badly on your customer at all. They were just going through what we all go through, dealing with everyday problems as they arise. What counts is how you dealt with those problems.

After all, this is a “success story” we’re talking about. If you hadn’t successfully resolved those issues, you wouldn’t be publishing it, would you? All you really have to do is emphasize the positive outcome: put your customer, company, and solution in the best possible light.

So give your audience a full, blow-by-blow description of the implementation process. Use as much detail as you can in the space available. Detail is the key to credibility. A detailed, engaging Implementation discussion in your case study will make the Results section that follows it all that more believable.

Take-Away Points

1. Case study readers (your prospects) want to know exactly:

a. What was required to implement your solution

b. How your company helped your customer overcome any obstacles encountered

2. Every case study should include a detailed, description of the system implementation.

3. Revealing challenges in the implementation process actually builds credibility and makes both your story and your product more compelling to technology buyers.

4. It’s not the problems but how you dealt with them that matters.

Need help crafting a compelling case study – one that includes a description of the implementation and follows the rules of good storytelling? Contact CopyEngineer by email at [email protected] .

Contact CopyEngineer

(See our  privacy policy )

OR Click here for more information.

Free Report

Not ready to talk about a new copywriting project just yet contact me anyway to get your free copy of my latest special report:.

10 Common Mistakes That Kill White Paper ROI How to Avoid Them and Generate More Leads

And don’t forget to sign up for my FREE e-zine,  Technical Response . Not only will you receive tips for better content marketing and lead generation. You’ll also receive a copy of my white paper, How to Plan a White Paper: A Proven 7-Step Process for Minimizing Headaches and Maximizing ROI , absolutely FREE. Subscribe now!

• Open access
• Published: 16 February 2022

The “case” for case studies: why we need high-quality examples of global implementation research

• Blythe Beecroft   ORCID: orcid.org/0000-0002-6254-421X 1 ,
• Rachel Sturke 1 ,
• Gila Neta 2 &
• Rohit Ramaswamy 3  

Implementation Science Communications volume  3 , Article number:  15 ( 2022 ) Cite this article

6 Citations

19 Altmetric

Metrics details

Rigorous and systematic documented examples of implementation research in global contexts can be a valuable resource and help build research capacity. In the context of low- and middle-income countries (LMICs), there is a need for practical examples of how to conduct implementation studies. To address this gap, Fogarty’s Center for Global Health Studies in collaboration with the Cincinnati Children's Hospital Medical Center and the National Cancer Institute is commissioning a collection of implementation science case studies in LMICs that describe key components of conducting implementation research, including how to select, adapt, and apply implementation science models, theories, and frameworks to these settings; develop and test implementation strategies; and evaluate implementation processes and outcomes. The case studies describe implementation research in various disease areas in LMICs around the world. This commentary highlights the value of case study methods commonly used in law and business schools as a source of “thick” (i.e., context-rich) description and a teaching tool for global implementation researchers. It addresses the independent merit of case studies as an evaluation approach for disseminating high-quality research in a format that is useful to a broad range of stakeholders. This commentary finally describes an approach for developing high-quality case studies of global implementation research, in order to be of value to a broad audience of researchers and practitioners.

Peer Review reports

Contributions to the literature

Reinforcing the need for “thick” (i.e., context-rich) descriptions of implementation studies

Highlighting the utility of case studies as a dissemination strategy for researchers, practitioners, and policymakers

Articulating the value of detailed case studies as a teaching tool for global implementation researchers

Describing a method for developing high-quality case studies of global implementation research

Research capacity for implementation science remains limited in low- and middle-income countries (LMICs). Various stakeholders, including NIH-funded implementation researchers and practitioners, often inquire about how to apply implementation science methods and have requested additional resources and training to support implementation capacity building. This is in part due to a dearth of practical examples for both researchers and practitioners of how to select, adapt, and apply implementation science models, theories, and frameworks to these settings; how to evaluate implementation processes and outcomes; and how to develop and test implementation strategies. The need for detailed documentation of implementation research in all settings has been well established, and guidelines for documentation of implementation research studies have been created [ 1 , 2 ]. But the mere availability of checklists and guidelines in and of themselves does not result in comprehensive documentation that is useful for learning, as has been pointed out by many systematic reviews of implementation science and quality improvement studies ([ 3 , 4 ]). It has also been observed that documentation alone is not enough, and there is a need for mentors to translate abstract theories into context-appropriate research designs and practice approaches [ 5 ]. Because of the especially acute shortage of mentors and coaches in LMIC settings, we propose that documentation with “thick” descriptions that go beyond checklists and guidelines are needed to make the field more useful to emerging professionals [ 6 ]. We suggest that the case study method intended to “explore the space between the world of theory and the experience of practice” [ 7 ] that has been used successfully for over a century by law and business schools as a teaching aid can be of value to develop detailed narratives of implementation research projects. In this definition, we are not referring to the case study as a qualitative research method [ 8 ], but as a rich and detailed method of retrospective documentation to aid teaching, practice, and research. In this context, our case studies are akin to “single-institution or single-patient descriptions” [ 9 ] called “case reports” or “case examples” in other fields. As these terms are rarely used in global health, we have used the words “case studies” in this paper but reiterate that they do not refer to case study research designs.

Fogarty’s Center for Global Health Studies (CGHS) in collaboration with the Cincinnati Children's Hospital Medical Center and the National Cancer Institute (NCI) is commissioning a collection of implementation science case studies that describe implementation research focusing on various disease areas in different (LMIC) contexts around the world. These case study descriptions will provide guidance on the process of conducting implementation science studies and will highlight the impact these studies have had on practice and policy in global health contexts. This brief note makes a case for using case studies to document and disseminate implementation research, describes the CGHS approach to case study development and poses evaluation questions that need to be answered to better understand the utility of case studies. This effort is intended to develop a set of useful examples for LMIC researchers, practitioners, and policymakers, but also to assess and improve the use of case studies as a tested documentation methodology in implementation research.

The “case” for case studies

A preliminary landscape analysis of the field conducted by CGHS found that there are not many descriptions of global implementation science projects in a case study format in the peer-reviewed or gray literature, and those that exist are embedded in the content of academic teaching materials. There is not a cohesive collection, especially relating to health, that illustrates how implementation research has been conducted in varied organizations, countries, or disease areas. This new collection will add value in three different ways: as a dissemination strategy, as a tool for capacity building, and as a vehicle for stimulating better research.

Case studies as a dissemination strategy

Case studies have independent merit as an evaluation approach for disseminating high-quality research in a format that is useful to a broad range of stakeholders. The Medical Research Council (MRC) has recommended process evaluation as a useful approach to examine complex implementation, mechanisms of impact, and context [ 10 ]. Guidelines on documentation of implementation recommend that researchers should provide “detailed descriptions of interventions (and implementation strategies) in published papers, clarify assumed change processes and design principles, provide access to manuals and protocols that provide information about the clinical interventions or implementation strategies, and give detailed descriptions of active control conditions” [ 1 ]. Case studies can be thought of as a form of post hoc process evaluation, to disseminate how the delivery of an intervention is achieved, the mechanisms by which implementation strategies produce change, or how context impacts implementation and related outcomes.

Case studies as a capacity building tool

In addition, case studies can address the universal recognition of the need for more capacity building in Implementation Science , especially in LMIC settings. Case studies have been shown to address common pedagogical challenges in helping students learn by allowing students to dissect and explore limitations, adaptations, and utilization of theories, thereby creating a bridge between theories presented in a classroom and their application in the field [ 11 ]. A recent learning needs assessment for implementation researchers, practitioners, and policymakers in LMICs conducted by Turner et al. [ 12 ] reflected a universal consensus on the need for context-specific knowledge about how to apply implementation science in practice, delivered in an interactive format supported by mentorship. A collection of case studies is a valuable and scalable resource to meet this need.

Using case studies to strengthen implementation research

Descriptions of research using studies can illustrate not just whether implementation research had an impact on practice and policy, but how, why, under what circumstances, and to whom, which is the ultimate goal of generating generalizable knowledge about how to implement. Using diverse cases to demonstrate how a variety of research designs have been used to answer complex implementation questions provides researchers with a palette of design options and examples of their use. A framework developed by Minary et al. [ 13 ] illustrates the wide variety of research designs that are useful for complex interventions, depending on whether the emphasis is on internal and external validity or whether knowledge about content and process or about outcomes is more important. A collection of case studies would be invaluable to researchers seeking to develop appropriate designs for their work. In addition, the detailed documentation provided through these case descriptions will hopefully motivate researchers to document their own studies better using the guidelines described earlier.

Developing and testing the case study creation process: the CGHS approach

Writing case studies that satisfy the objectives described above is an implementation science undertaking in itself that requires the engagement of a variety of stakeholders and planned implementation strategies. The CGHS team responsible for commissioning the case studies began this process in 2017 and followed the approach detailed below to test the process of case study development.

Conducted 25+ consultations with various implementation science experts on gaps in the field and the relevance of global case studies

Convened a 15-member Steering Committee Footnote 1 of implementation scientists with diverse expertise, from various academic institutions and NIH institutes to serve as technical experts and to help guide the development and execution of the project

Developed a case study protocol in partnership with the Steering Committee to guide the inclusion of key elements in the case studies

Commissioned two pilot cases Footnote 2 to assess the feasibility and utility of the case study protocol and elicited feedback on the writing experience and how it could be improved as the collection expands

Led an iterative pilot writing process where each case study writing team developed several drafts, which were reviewed by CGHS staff and a designated member of the Steering Committee

Truncated and adjusted the protocol in response to input from the pilot case study authors teams

Developed a comprehensive grid with the Steering Committee, outlining the key dimensions of implementation science that are significant and would be important areas of focus for future case studies. The grid will be used to evaluate potential case applicants and is intended to help foster diversity of focus and content, in addition to geography

Implementing the process: the call for case studies

In March of 2021, CGHS issued a closed call for case studies to solicit applications from a pool of researchers. Potential applicants completed the comprehensive grid in addition to a case study proposal. Applicants will go through a three-tier screening and review process. CGHS will initially screen the applications for completeness to ensure all required elements are present. Each case study application will then be reviewed by two Steering Committee members for content and scientific rigor and given a numerical score based on the selection criteria. Finally, the CGHS team will screen the applications to ensure diversity of implementation elements, geography, and disease area. Approximately 10 case studies will be selected for development in an iterative process. Each case team will present their case drafts to the Steering Committee, which will collectively workshop the drafts in multiple sittings, drawing on the committee’s implementation science expertise. Once case study manuscripts are accepted by the Steering Committee, they will be submitted to Implementation Science Communications for independent review by the journal. CGHS intends for the case studies to be published collectively, but on a rolling basis as they are accepted for publication.

Future research: evaluating the effectiveness of the case study approach

This commentary has put forth arguments for the potential value of case studies for documenting implementation research for researchers, practitioners, and policymakers. Case studies not only provide a way to underscore how implementation science can advance practice and policy in LMICs, but also offer guidance on how to conduct implementation research tailored to global contexts. However, there is little empirical evidence about the validity of these arguments. The creation of this body of case studies will allow us to study why, how, how often, and by whom these case studies are used. This is a valuable opportunity to learn and use that information to better inform future use of this approach as a capacity-building or dissemination strategy.

Conclusions

Similar to their use in law and business, case study descriptions of implementation research could be an important mechanism to counteract the paucity of training programs and mentors to meet the demands of global health researchers. If the evaluation results indicate that the case study creation process produces useful products that enhance learning to improve future implementation research, a mechanism needs to be put in place to create more case studies than the small set that can be generated through this initiative. There will be a need to create a set of documentation guidelines that complement those that currently exist and a mechanism to solicit, review, publish, and disseminate case studies from a wide variety of researchers and practitioners. Journals such as Implementation Science or Implementation Science Communications can facilitate this effort by either creating a new article type or by considering a new journal with a focus on rigorous and systematic case study descriptions of implementation research and practice. An example that could serve as a guide is BMJ Open Quality , which is a peer-reviewed, open-access journal focused on healthcare improvement. In addition to original research and systematic reviews, the journal publishes two article types: Quality Improvement Report and Quality Education Report to document healthcare quality improvement programs and training. The journal offers resources for authors to document their work rigorously. Recently, a new journal titled BMJ Open Quality South Asia has been released to disseminate regional research. We hope that our efforts in sponsoring and publishing these cases, and in setting up a process to support their creation, will make an important contribution to the field and become a mechanism for sharing knowledge that accelerates the growth of implementation science in LMIC settings.

Availability of data and materials

Not applicable.

Rohit Ramaswamy, CCHMC, Gila Neta, NCI NIH, Theresa Betancourt, BC, Ross Brownson, WASU, David Chambers, NCI NIH, Sharon Straus, University of Toronto, Greg Aarons, UCSD, Bryan Weiner, UW, Sonia Lee, NICHD NIH, Andrea Horvath Marques, NIMH NIH, Susannah Allison, NIMH NIH, Suzy Pollard, NIMH NIH, Chris Gordon, NIMH NIH, Kenny Sherr, UW, Usman Hamdani, HDR Foundation Pakistan, Linda Kupfer, FIC NIH

The first pilot case was led by the Human Development Research Foundation (HDRF) in Pakistan and examines scaling up evidenced-based care for children with developmental disorders in rural Pakistan. The second pilot was led by Boston College and investigates alternate delivery platforms and implementation models for bringing evidence-based behavioral Interventions to scale for youth facing adversity in Sierra Leone to close the mental health treatment gap.

Abbreviations

Low- and middle-income countries

Center for Global Health Studies

National Cancer Institute

Medical Research Council

Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implementation Sci. 2013;8(139) [cited 2021 May];. Available from: https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-8-139 .

Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, et al. for the StaRI group. Standards for Reporting Implementation Studies (StaRI) statement. BMJ. 2017;356:i6795 [cited 2021Sept]. Available from: https://www.bmj.com/content/356/bmj.i6795 .

Article   Google Scholar  

Brouwers MC, De Vito C, Bahirathan L, Carol A, Carroll JC, Cotterchio M, et al. What implementation interventions increase cancer screening rates? A systematic review. Implementation Sci. 2011; [cited 2021 Sept];6:111. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3197548/ .

Nadeem E, Gleacher A, Beidas RS. Consultation as an implementation strategy for evidence-based practices across multiple contexts: unpacking the black box. Adm Policy Ment Health. 2013;40(6):439–50. [cited 2021 Sept]. Available from: https://link.springer.com/article/10.1007%2Fs10488-013-0502-8 .

Darnell D, Dorsey CN, Melvin A, Chi J, Lyon AR, Lewis CC. A content analysis of dissemination and implementation science resource initiatives: what types of resources do they offer to advance the field? Implement Sci. 2017;12(1):137 [cited 2021 Sept];. Available from: https://implementationscience.biomedcentral.com/articles/10.1186/s13012-017-0673-x .

Geertz C. Thick description: toward an interpretive theory of culture In: The interpretation of cultures: selected essays. New York: Basic Books; 1973. p. 3–32.

Breslin M, Buchanan R. On the case study method of research and teaching in design. Design Issues. 2008;24(1):36–40 [cited 2021 Sept]. Available from: http://www.jstor.org/stable/25224148 .

Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach. BMC Med Res Methodol. 2011;11:100. [cited 2021 Sept]. Available from. https://doi.org/10.1186/1471-2288-11-100 .

Article   PubMed   PubMed Central   Google Scholar  

Alpi KM, Evans JJ. Distinguishing case study as a research method from case reports as a publication type. J Med Libr Assoc. 2019;107(1):1–5 Jan [cited 2021 Sept]. Available from: https://jmla.pitt.edu/ojs/jmla/article/view/615 .

Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258. March [cited 2021 May]. Available from: https://www.bmj.com/content/350/bmj.h1258 . https://doi.org/10.1136/bmj.h1258 .

Velenchik AD. The case method as a strategy for teaching policy analysis to undergraduates. J. Econ. Educ. 1995;26(1):29–38.

Turner MW, Bogdewic S, Agha E, Blanchard C, Sturke R, Pettifor A, et al. Learning needs assessment for multi-stakeholder implementation science training in LMIC settings: findings and recommendations. Implement Sci Commun. 2021;2(134) [cited 2021 May]. Available from: https://implementationsciencecomms.biomedcentral.com/articles/10.1186/s43058-021-00238-2 .

Minary L, Trompette J, Kivits J, Cambon L, Tarquinio C, Alla F. Which design to evaluate complex interventions? Toward a methodological framework through a systematic review. BMC Med Res Methodol. 2019;19(92) [cited 2021 May]. Available from: https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-019-0736-6 .

Download references

Acknowledgements

The findings and conclusions in this manuscript are those of the authors and do not necessarily represent any official position or policy of the US National Institutes of Health or the US Department of Health and Human Services or any other institutions with which authors are affiliated.

Author information

Authors and affiliations.

Fogarty International Center, US National Institutes of Health, Bethesda, USA

Blythe Beecroft & Rachel Sturke

National Cancer Institute, US National Institutes of Health, Bethesda, USA

Cincinnati Children’s Hospital Medical Center, Cincinnati, USA

Rohit Ramaswamy

You can also search for this author in PubMed   Google Scholar

Contributions

BB, RS, and GN contributed to the conceptualization of the manuscript with leadership from RR. BB and RS drafted the main text. RR and GN reviewed and contributed additional content to further develop the text. All authors have read and agreed to the contents of the final draft of the manuscript.

Corresponding author

Correspondence to Blythe Beecroft .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Beecroft, B., Sturke, R., Neta, G. et al. The “case” for case studies: why we need high-quality examples of global implementation research. Implement Sci Commun 3 , 15 (2022). https://doi.org/10.1186/s43058-021-00227-5

Download citation

Received : 05 August 2021

Accepted : 27 September 2021

Published : 16 February 2022

DOI : https://doi.org/10.1186/s43058-021-00227-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Case studies
• Global implementation science

Implementation Science Communications

ISSN: 2662-2211

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Open access
• Published: 01 July 2022

Challenges to implementing artificial intelligence in healthcare: a qualitative interview study with healthcare leaders in Sweden

• Lena Petersson 1 ,
• Ingrid Larsson 1 ,
• Jens M. Nygren 1 ,
• Per Nilsen 1 , 2 ,
• Margit Neher 1 , 3 ,
• Julie E. Reed 1 ,
• Daniel Tyskbo 1 &
• Petra Svedberg 1  

BMC Health Services Research volume  22 , Article number:  850 ( 2022 ) Cite this article

55 Citations

30 Altmetric

Metrics details

Artificial intelligence (AI) for healthcare presents potential solutions to some of the challenges faced by health systems around the world. However, it is well established in implementation and innovation research that novel technologies are often resisted by healthcare leaders, which contributes to their slow and variable uptake. Although research on various stakeholders’ perspectives on AI implementation has been undertaken, very few studies have investigated leaders’ perspectives on the issue of AI implementation in healthcare. It is essential to understand the perspectives of healthcare leaders, because they have a key role in the implementation process of new technologies in healthcare. The aim of this study was to explore challenges perceived by leaders in a regional Swedish healthcare setting concerning the implementation of AI in healthcare.

The study takes an explorative qualitative approach. Individual, semi-structured interviews were conducted from October 2020 to May 2021 with 26 healthcare leaders. The analysis was performed using qualitative content analysis, with an inductive approach.

The analysis yielded three categories, representing three types of challenge perceived to be linked with the implementation of AI in healthcare: 1) Conditions external to the healthcare system; 2) Capacity for strategic change management; 3) Transformation of healthcare professions and healthcare practice.

Conclusions

In conclusion, healthcare leaders highlighted several implementation challenges in relation to AI within and beyond the healthcare system in general and their organisations in particular. The challenges comprised conditions external to the healthcare system, internal capacity for strategic change management, along with transformation of healthcare professions and healthcare practice. The results point to the need to develop implementation strategies across healthcare organisations to address challenges to AI-specific capacity building. Laws and policies are needed to regulate the design and execution of effective AI implementation strategies. There is a need to invest time and resources in implementation processes, with collaboration across healthcare, county councils, and industry partnerships.

Peer Review reports

The use of artificial intelligence (AI) in healthcare can potentially enable solutions to some of the challenges faced by healthcare systems around the world [ 1 , 2 , 3 ]. AI generally refers to a computerized system (hardware or software) that is equipped with the capacity to perform tasks or reasoning processes that we usually associate with the intelligence level of a human being [ 4 ]. AI is thus not one single type of technology but rather many different types within various application areas, e.g., diagnosis and treatment, patient engagement and adherence, and administrative activities [ 5 , 6 ]. However, when implementing AI technology in practice, certain problems and challenges may require an optimization of the method in combination with the specific setting. We may therefore define AI as complex sociotechnical interventions as their success in a clinical healthcare setting depends on more than the technical performance [ 7 ]. Research suggests that AI technology may be able to improve the treatment of many health conditions, provide information to support decision-making, minimize medical errors and optimize care processes, make healthcare more accessible, provide better patient experiences and care outcomes as well as reduce the per capita costs of healthcare [ 8 , 9 , 10 ]. Even if the expectations for AI in healthcare are great [ 2 ], the potential of its use in healthcare is far from having been realized [ 5 , 11 , 12 ].

Most of the research on AI in healthcare focuses heavily on the development, validation, and evaluation of advanced analytical techniques, and the most significant clinical specialties for this are oncology, neurology, and cardiology [ 2 , 3 , 11 , 13 , 14 ]. There is, however, a current research gap between the development of robust algorithms and the implementation of AI systems in healthcare practice. The conclusion in newly published reviews addressing regulation, privacy and legal aspects [ 15 , 16 ], ethics [ 16 , 17 , 18 ], clinical and patient outcomes [ 19 , 20 , 21 ] and economic impact [ 22 ], is that further research is needed in a real-world clinical setting although the clinical implementation of AI technology is still at an early stage. There are no studies describing implementation frameworks or models that could inform us concerning the role of barriers and facilitators in the implementation process and relevant implementation strategies of AI technology [ 23 ]. This illustrates a significant knowledge gap on how to implement AI in healthcare practice and how to understand the variation of acceptance of this technology among healthcare leaders, healthcare professionals, and patients [ 14 ]. It is well established in implementation and innovation research that novel technologies, such as AI, are often resisted by healthcare leaders, which contributes to their slow and variable uptake [ 13 , 24 , 25 , 26 ]. New technologies often fail to be implemented and embedded in practice because healthcare leaders do not consider how they fit with or impact existing healthcare work practices and processes [ 27 ]. Although, understanding how AI technologies should be implemented in healthcare practice is unexplored.

Based on literature from other scientific fields, we know that the leaders’interest and commitment is widely recognized as an important factor for successful implementation of new innovations and interventions [ 28 , 29 ]. The implementation of AI in healthcare is thus supposed to require leaders who understand the state of various AI systems. The leaders have to drive and support the introduction of AI systems, the integration into existing or altered work routines and processes, and how AI systems can be deployed to improve efficiency, safety, and access to healthcare services [ 30 , 31 ]. There is convincing evidence from outside the healthcare field of the importance of leadership for organizational culture and performance [ 32 ], the implementation of planned organizational change [ 33 ], and the implementation and stimulation of organizational innovation [ 34 ]. The relevance of leadership to implementing new practices in healthcare is reflected in many of the theories, frameworks, and models used in implementation research that analyses barriers to and facilitators of its implementation [ 35 ]. For example, Promoting Action on Research Implementation in Health Services [ 36 ], Consolidated Framework for Implementation Research (CFIR) [ 37 ], Active Implementation Frameworks [ 38 ], and Tailored Implementation for Chronic Diseases [ 39 ] all refer to leadership as a determinant of successful implementation. Although these implementation models are available and frequently used in healthcare research, they are highly abstract and not tailored to the implementation of AI systems in healthcare practices. We thus do not know if these models are applicable to AI as a socio-technical system or if other determinants are important for the implementation process. Likewise, based on a new literature study, we found no AI-specific implementation theories, frameworks, or models that could provide guidance for how leaders could facilitate the implementation and realize the potential of AI in healthcare [ 23 ]. We thus need to understand what the unique challenges are when implementing AI in healthcare practices.

Research on various types of stakeholder perspectives on AI implementation in healthcare has been undertaken, including studies involving professionals [ 40 , 41 , 42 , 43 ], patients [ 44 ], and industry partners [ 42 ]. However, very few studies have investigated the perspectives of healthcare leaders. This is a major shortcoming, given that healthcare leaders are expected to have a key role in the implementation and use of AI for the development of healthcare. Petitgand et al.’s study [ 45 ] serves as a notable exception. They interviewed healthcare managers, providers, and organizational developers to identify barriers to integrating an AI decision-support system to enhance diagnostic procedures in emergency care. However, the study did not focus on the leaders’ perspectives, and the study was limited to one particular type of AI solution in one specific care department. Our present study extends beyond any specific technology and encompasses the whole socio-technical system around AI technology. The present study thus aimed to explore challenges perceived by leaders in a regional Swedish healthcare setting regarding implementation of AI systems in healthcare.

This study took an explorative qualitative approach to understanding healthcare leaders’ perceptions in contexts in which AI will be developed and implemented. The knowledge generated from this study will inform the development of strategies to support an AI implementation and help avoid potential barriers. The analysis was based on qualitative content analysis, with an inductive approach [ 46 ]. Qualitative content analysis is widely used in healthcare research [ 46 ] to find similarities and differences in the data, in order to understand human experiences [ 47 ]. To ensure trustworthiness, the study is reported in accordance with the Consolidated Criteria for Reporting Qualitative Research 32‐item checklist [ 48 ].

The study was conducted in a county council (also known as “region”) in the south of Sweden. The Swedish healthcare system is publicly financed based on local taxation; residents are insured by the state and there is a vision that healthcare should be equally accessible across the population. Healthcare responsibility is decentralized to 21 county councils, whose responsibilities include healthcare provision and promotion of good health for citizens.

The county council under investigation has since 2016 invested financial, personnel and service resources to enable agile analysis (based on machine learning models) of clinical and administrative data of patients in healthcare [ 49 , 50 ]. The ambition is to gain more value from the data, utilizing insights drawn from machine learning on healthcare data to make facts-based decisions on how healthcare is managed, organized, and structured in routines and processes. The focus is thus on overall issues around management, staffing, planning and standardization for optimization of resource use, workflows, patient trajectories and quality improvement at system level. This includes several layers within the socio-technical ecosystem around the technology, dealing with: a) generating, cleaning, and labeling data, b) developing models, verifying, assuring, and auditing AI tools and algorithms, c) incorporating AI outputs into clinical decisions and resource allocation, and d) the shaping of new organizational structures, roles, and practices. Given that AI thus extends beyond any specific technology and encompasses the whole socio-technical system around the technology, in the context of this article, it is hereafter referred to generically as ‘AI systems’. We deliberately sought to understand the broad perspectives on healthcare leaders in a region that has a high level of support for AI developments and our study thus focuses on the potential of a wide range of AI systems that could emerge from the regional investments, rather than a specific AI application or AI algorithms.

Participants

Given the focus on understanding healthcare leaders’ perceptions, we purposively recruited leaders who were in a position to potentially influence the implementation and use of AI systems in relation to the setting described above. To achieve potential variability, these leaders belonged to three groups: politicians at the highest county council level, managers at various levels, such as the hospital director, manager for primary care, manager for knowledge and evidence, head of research and development center, and quality developers and strategists with responsibilities for strategy-based work at county council level or development work in various divisions in the county council healthcare organization.

The ambition was to include leaders who had a range of experiences, interests and with different mandates and responsibilities in relation to funding, running, and sustaining the implementation of AI systems in practice. A sample of 28 healthcare leaders was invited through snowball recruitment; two declined and 26 agreed to participate (Table 1 ). This sample comprised five individuals originally identified on the basis of their knowledge and insights. They were interviewed and they then identified and suggested other leaders to interview.

Data collection

Individual semi-structured interviews were conducted between October 2020 and May 2021 via phone or video communication by one of the authors (LP or DT). We start from a broad perspective on AI focusing on healthcare leaders’ perceptions bottom-up and not on the views of AI experts or healthcare professionals who work with specific AI algortihms in clinical practice. The interviews were based on an interview guide, structured around: 1) the roles and previous experiences of the informants regarding the application of AI systems in practice, 2) the opportunities and problems that need to be considered to support implementation of AI systems, 3) beliefs and attitudes towards the possibilities of using AI systems to support healthcare improvements, and 4) the obstacles, opportunities and facilitating factors that need to be considered to enable AI systems to fit into existing processes, methods and systems. The interview guide was thus based on important factors previously identified in terms of implementing technology in healthcare [ 51 , 52 ]. Interviews lasted between 30 and 120 min, with a total length of 23 h and 49 min and were audio-recorded.

Data analysis

An inductive qualitative content analysis [ 46 ] was used to analyze the data. First, the interviews were transcribed verbatim and read several times by the first (LP) and second (IL) authors, to gain familiarity. Then, the first (LP) and second (IL) authors conducted the initial analyses of the interviews, by identifying and extracting meaning units and/or phrases with information relevant to the object of the study. The meaning units were then abstracted into codes, subcategories, and categories. The analytical process was discussed continuously between authors (LP, IL, JMN, PN, MN, PS). Finally, all authors, who are from different disciplines, reviewed and discussed the analysis to increase the trustworthiness and rigour of the analysis. To further strengthen the trustworthiness, the leaders’ quotations used in this paper were translated from Swedish to English by a native English-speaking professional proofreader and were edited only slightly to improve readability.

Three categories consisting of nine sub-categories emerged from the analysis of the interviews with the healthcare leaders (Fig.  1 ). Conditions external to the healthcare system concern various exogenous conditions and circumstances beyond the direct control of the healthcare system that the leaders believed could affect AI implementation. Capacity for strategic change management reflects endogenous influences and internal requirements related to the healthcare system that the leaders suggested could pose challenges to AI implementation. Transformation of healthcare professions and healthcare practice concerns challenges to AI implementation observed by the leaders, in terms of how AI might change professional roles and relations and its impact on existing work practices and routines.

Categories and subcategories

Conditions external to the healthcare system

Addressing liability issues and legal information sharing.

The healthcare leaders described the management of existing laws and policies for the implementation of AI systems in healthcare as a challenge and an issue that was essential to address. According to them, the existing laws and policies have not kept pace with technological developments and the organization of healthcare in today’s society and need to be revised to ensure liability.

The accountability held among individuals, organizations, and AI systems regarding decisions based on support from an AI algorithm was perceived as a risk and an element that needs to be addressed. However, accountability is not addressed in existing laws, which were perceived by the leaders to present problematic uncertainties in terms of responsibilities. They raised concerns about where responsibilities lie in relation to decisions made by AI algorithms, such as when an AI algorithm run in one part of the system identifies actions that should be taken in another part of the system. For example, if a patient is given AI-based advice from a county council-operated patient portal for triaging suggesting self-care, and the advice instead should have been to visit the emergency department, who has the responsibility, is it the AI system itself, the developers of the system or the county council. Additionally, concerns were raised about accountability, if it turns out that the advice was not accurate.

The issue of accountability is a very difficult one. If I agree with what doctor John (AI systems) recommended, where does the burden of proof lie? I may have looked at this advice and thought that it worked quite well. I chose to follow this advice, but can I blame Doctor John? The legislation is a risk that we have to deal with. Leader 7.

Concerns were raised as to how errors would be handled when AI systems contributed to decision making, highlighting the need for clear laws and policies. The leaders emphasized that, if healthcare professionals made erroneous decisions based on AI systems, they could be reported to the Patients Advisory Committee or have their medical license revoked. This impending threat could lead to a stressful situation for healthcare professionals. The leaders expressed major concerns about whether AI systems would be support systems for healthcare professionals’ decisions or systems that could take automated and independent decisions. They believed based on the latter interpretation that there would be a need for changes in the laws before they could be implemented in practice. Nevertheless, some leaders anticipated a development where some aspects of care could be provided without any human involvement.

If the legislation is changed so that the management information can be automated, that is to say that they start acting themselves, but they’re not allowed to do that yet. It could, however, be so that you open an app in a few years’ time, then you furnish the app with the information that it needs about your health status. Then the app can write a prescription for medication for you, because it has all the information that is needed. That is not allowed at present, because the judicial authority still need an individual to blame when something goes wrong. But even that aspect will be gradually developed. Leader 2.

According to the leaders, legislation and policies also constituted obstacles to the foundation in the implementation of AI systems in healthcare: collecting, using, merging, and analyzing patient information. The limited opportunities to legally access and share information about patients within and between organizations were described as a crucial obstacle in implementing and using AI systems. Another issue was the legal problems when a care provider wanted to merge information about patients from different providers, such as the county council and a municipality. For this to take place, it was perceived that a considerable change of the laws regulating the possibilities of sharing information across different care providers would be required. Additionally, there are challenges in the definition of personal data in laws regulating personal integrity and in the risk of individuals being identified when the data is used for computerized advanced analytics. The law states that it is not legal to share personal data, but the boundaries of what is constituted by personal data in today’s society are changing, due to the increasing amounts of data and opportunities for complex and intelligent analysis.

You are not allowed to share any personal information. No, we understand that but what is personal information and when is personal information no longer personal information? Because legally speaking it is definitely not just the case of removing the personal identity number and the name, as a computer can still identify who you are at an individual level. When can it not do that? Leader 2.

Thus, according to the healthcare leaders, laws and regulations presented challenges for an organization that want to implement AI systems in healthcare practice, as laws and regulations have different purposes and oppose each other, e.g., the Health and Medical Services Act, the Patient Act and the Secrecy Act. Leaders described how outdated laws and regulations are handled in healthcare practice, by stretching current regulations and attempts to contribute to changing laws . They aimed to not give up on visions and ideas, but to try to find gaps in existing laws and to use rather than break the laws. When possible, another way to approach this was to try to influence decision-makers on the national political level to change the laws. The leaders reported that civil servants and politicians in the county council do this lobbying work in different contexts, such as the parliament or the Swedish Association of Local Authorities and Regions (SALAR).

We discuss this regularly with our members of parliament with the aim of influencing the legislative work towards an enabling of the flow of information over boundaries. It’s all a bit old-fashioned. Leader 16.

Complying with standards and quality requirements

The healthcare leaders believed it could be challenging to follow standardized care processes when AI systems are implemented in healthcare. Standardized care processes are an essential feature that has contributed to development and improved quality in Swedish healthcare. However, some leaders expressed that the implementation of AI systems could be problematic because of uncertainties regarding when an AI algorithm is valid enough to be a part of a standardized care process. They were uncertain about which guarantees would be required for a product or service before it would be considered “good enough” and safe to use in routine care. An important legal aspect for AI implementation is the updated EU regulation for medical devices (MDR) that came into force in May 2021. According to one of the leaders, this regulation could be problematic for small innovative companies, as they are not used to these demands and will not always have the resources needed to live up to the requirements. Therefore, the leaders perceived that the county council should support AI companies to navigate these demands, if they are to succeed in bringing their products or services to implementation in standardized care processes.

We have to probably help the narrow, supersmart and valuable ideas to be realized, so that there won’t be a cemetery of ideas with things that could have been good for our patients, if only the companies had been given the conditions and support to live up to the demands that the healthcare services have and must have in terms of quality and security. Leader 2.

Integrating AI-relevant learning in higher education for healthcare staff

The healthcare leaders described that changes needed to be made in professional training, so that new healthcare professionals would be prepared to use digital technology in their practical work. Some leaders were worried that basic level education for healthcare professionals, such as physicians, nurses, and assistant nurses has too little focus on digital technology in general, and AI systems in particular. They stated that it is crucial that these educational programs are restructured and adapted to prepare students for the ongoing digitalization of the healthcare sector. Otherwise, recently graduated healthcare professionals will not be ready to take part in utilizing and implementing new AI systems in practice.

I am fundamentally quite concerned that our education, mainly when it comes to the healthcare services. Both for doctors and nurses and also assistant nurses for that matter. That it isn’t sufficiently proactive and prepare those who educate themselves for what will come in the future. // I can feel a certain concern for the fact that our educations do not actually sufficiently prepare our future co-workers for what everybody is talking now about that will take place in the healthcare services. Leader 15.

Capacity for strategic change management

Developing a systematic approach to ai implementation.

The healthcare leaders described that there is a need for a systematic approach and shared plans and strategies at the county council level, in order to meet the challenge of implementing AI systems in practice. They recognized that it will not be successful if the change is built on individual interests, instead of organizational perspectives. According to the leaders, the county council has focused on building the technical infrastructure that enables the use of AI algorithms. The county council have tried to establish a way of working with multi-professional teams around each application area for AI-based analysis. However, the leaders expressed that it is necessary to look beyond the technology development and plan for the implementation at a much earlier stage in the development process. They believed that their organization generally underestimated the challenges of implementation in practice. Therefore, the leaders believed that it was essential that the politicians and the highest leadership in the county council both support and prioritize the change process. This requires an infrastructure for strategic change management together with clear leadership that has the mandate and the power to prioritize and support both development of AI systems and implementation in practice. This is critical for strategic change to be successful.

If the County Council management does not believe in this, then nothing will come of it either, the County Council management have to indicate in some way that this is a prioritized issue. It is this we are going to work with, then it’s not sufficient for a single executive director who pursues this and who thinks it’s interesting. It has to start at the top and then filter right through, but then the politicians have to also believe in this and think that it’s important. Leader 4.

Additionally, the healthcare leaders experienced that there was increasing interest among unit managers within the organization in using data for AI-based analysis and that there might be a need to make more prioritizations of requests for data analysis in the future. The leaders expressed that it would not be enough to simply have a shared core facility supporting this. Instead, management at all levels should also be involved and active in prioritization, based on their needs. They also perceived that the implementation of AI systems will demand skilled and structured change management that can prioritize and that is open to new types of leadership and decision-making processes. Support for innovative work will be needed, but also caution so that change does not proceed too quickly and is sufficiently anchored among the staff. The implementation of AI systems in healthcare was anticipated to challenge old routines and replace them with new ones, and that, as a result, would meet resistance from the staff. Therefore, a prepared plan at the county council level was perceived to be required for the purpose of “anchoring” with managers at the unit level, so that the overall strategy would be aligned with the needs and views of those who would have to implement it and supported by the knowledge needed to lead the implementation work.

It’s in the process of establishing legitimacy that we have often erred, where we’ve made mistakes and mistakes and mistakes all the time, I’ve said. That we’re not at the right level to make the decisions and that we don’t follow up and see that they understand what it’s about and take it in. It’s from the lowest manager to the middle manager to executive directors to politicians, the decisions have to have been gained legitimacy otherwise we’ll not get the impetus. Leader 21.

The leaders believed that it was essential to consider how to evaluate different parts of the implementation process. They expressed that method development is required within the county council, because, at the moment, there is a lack of knowledge and guidelines on how to evidence-base the use of AI systems in practice. There will be a need for a support organization spanning different levels within the county council, to guide and supervise units in the systematic evaluation of AI implementations. There will also be a need for quantitative evaluation of the clinical and organizational effects and qualitative assessment that focuses on how healthcare professionals and patients experience the implementation. Additionally, validation and evaluation of AI algorithms will be needed, both before they can be used in routine care, and afterwards, to provide evidence of quality improvements and optimizations of resources.

I believe that one needs to get an approval in some way, perhaps not from the Swedish Medical Products Agency, but the AI Agency or something similar. I don’t know. The Swedish National Board of Health and Welfare or some agency needs to go in and check that it is a sufficiently good foundation that they have based this algorithm on. So that it can be approved for clinical use. Leader 10.

Furthermore, the leaders described a challenge around how the implementation of AI systems in practice could be sustainable and last over time. They expressed that the county council should develop strategies in the organization so that they are readied for sustainability and long-term implementation. At the same time, this is an area with fast development and high uncertainty about the future, and thus what AI systems and services will look like in five or ten years, and how healthcare professionals and patients will use them. This is a challenge and requires that both leaders and staff are prepared to adjust and change their ways of working during the implementation process, including continuous improvements and uptake, updating and evolution of technologies and work practices.

The rate of change where digitalization, technology, new technology and AI is concerned is so high and the rate of implementation is low, so this will entail that as soon as we are about to implement something then there is something else in the market that is better. So I think it’s important to dare to implement something that is a little further on in the future. Leader 13.

Ascertaining resources for AI implementation

The leaders emphasized the importance of training for implementation of AI systems in healthcare. The county council should provide customized training at the workplace and extra knowledge support for certain professions. This could result in difficult decisions regarding what and whom to prioritize. The leaders discussed whether there was a need to provide all staff with basic training on AI systems or if it would be enough to train some of them, such as quality developers, and provide targeted training for some healthcare professionals who are close to the implementation of the AI system at a care unit. Furthermore, the leaders described that the training had to be connected to implementing the AI system at a specific care unit, which could present a challenge for the planning and realization. They emphasized that it could be a waste of resources to educate the staff beforehand. They need to be educated in close connection to the implementation of a specific AI system in their workplace, which thus demands organizational resources and planning.

I think that we often make the mistake of educating first, and then you have to use it. But you have been educated, so now you should know this? Yes, but it is not until we use something that the questions arise. Leader 13.

There could also be a need for patient education and patient guidance, if they are to use AI systems for self-care or remote monitoring. Thus, it is vital to give all citizens the same opportunities to access and utilize new technical solutions in healthcare.

We treat all our patients equally now, everyone will receive the same invitation, and everyone will need to ring about their appointment, although 99% could really book and do this themselves. Then we should focus on that, and thus return the impetus and the power to the patient and the population for them to take care of this themselves to a greater extent. But then of course information is needed and that in turn needs intuitive systems. That is not something we are known for. Leader 14.

Many of the healthcare leaders found financial resources and time, especially the prioritization of time, to be critical to the implementation process of AI system. There is already time pressure in many care units, and it can be challenging to set aside time and other resources for the implementation.

Involving staff throughout the implementation process of AI systems

The healthcare leaders stated that anchoring and involving staff and citizens is crucial to the successfully implementation of AI systems. The management has to be responsible for the implementation process but also ensure that the staff are aware of and interested in the implementation, based on their needs. Involvement of the staff together with representatives from patient groups was considered key to successful implementation and to limit risks of perceiving the AI system as unnecessary and erroneously used. At the same time, the leaders described that it would be important for unit managers to “stand up” for the change that is required, if their staff questioned the implementation.

I think for example that if you’re going to make a successful implementation then you have to perhaps involve the co-workers. You can’t involve all of them, but a representative sample of co-workers and patients and the population who are part of it. // We mess it up time after time, and something comes that we have to implement with short notice. So we try to force it on the organization, so we forget that we need to get the support of the co-workers. Leader 4.

The propensity for change differs both among individuals and within the organization. According to the leaders, that could pose a challenge, since the support and needs differ between individuals. The motivational aspect could also vary between different actors, and some leaders claim that it is crucial to arouse curiosity among healthcare professionals. If the leaders are not motivated and do not believe that the change benefits them, implementation will not be successful. To increase healthcare professionals’ motivation and engagement, the value that will be created for the clinicians has to be made obvious, along with whether the AI system will support them in their daily work.

It has to be beneficial for the clinics otherwise it’s meaningless so to speak. A big risk with AI is that you work and work with data and then algorithms emerge that are sort of obvious. Everyone can do this. It’s why it’s important to have clinical staff in the small agile teams, that there really is a clinical benefit, this actually improves it. Leader 10.

Developing new strategies for internal and external collaboration

The healthcare leaders believed that there was a need for new forms of collaboration and communication within the county council, at both organizational and professional levels. Professionals need to interact with professions other than their own, thus enabling new teamwork and new knowledge. The challenge is for different groups to talk to each other, since they do not always have the same professional language. However, it was perceived that, when these kinds of team collaborations are successful, there will be benefits, such as automation of care processes that are currently handled by humans.

To be successful in getting a person with expert knowledge in computer science to talk to a person with expert knowledge in integrity legislation, to a one who has expert knowledge in the clinical care of a patient. Even if all of them go to work with exactly the same objective, that one person or a few people can live a bit longer or feel a bit better, then it’s difficult to talk with each other because they use essentially different languages. They don’t know much about what knowledge the other has, so just getting that altogether. Leader 2.

Leaders’ views the implementation of AI systems would require the involvement and collaboration of several departments in the county council across organizational boundaries, and with external actors. A perceived challenge was that half of the primary care units are owned by private care providers, where the county council has limited jurisdiction, which challenges the dissemination of common ways of working. Additionally, the organization in the county council and its boundaries might have to be reviewed to enable different professions to work together and interact on an everyday basis.

The complexity in terms of for example apps is very, very, very much greater, we see that now. Besides there being this app, so perhaps the procurement department must be involved, the systems administration must definitely be involved, the knowledge department must be involved and the digitalization department, there are so many and the finance department of course and the communication department, the system is thus so complex. Leader 9.

There was also consensus among the healthcare leaders that the county council should collaborate with companies in AI systems implementation and should not handle such processes on their own. An eco-system of actors working in AI systems implementation is required, who have shared goals for the joint work. The leaders expressed that companies must be supported and invited to collaborate within the county council’s organization at an early stage. In that way, pitfalls regarding legal or technical aspects can be discovered early in product development. Similar relations and dialogues are also needed with patients to succeed with implementation that is not primarily based on technical possibilities, but patients’ needs. Transparency is essential to patients’ awareness of AI systems’ functions and for the reliability in outcomes.

This is born out of a management philosophy, which is based on the principle of not being able to command everything oneself, one has to be humble, perceptive about not being able to do it. One needs to invite others to be there and help with the solution. Leader 16.

Transformation of healthcare professions and healthcare practices

Managing new roles in care processes.

The healthcare leaders described a need for new professions and professional roles in healthcare for AI systems implementation. All professional groups in today’s healthcare sector were expected to be affected by these changes, particularly the work unit managers responsible for daily work processes and the physicians accountable for the medical decisions. The leaders argued that the changes could challenge traditions, hierarchies, conventional professional roles and division of labour. There might be changes regarding the responsibilities for specific work tasks, changes in professional roles, a need for new professions that do not exist in today’s labour market and the AI systems might replace some work tasks and even professions. A change towards more combined positions at both the county council and a company or a university might also be a result of the development and implementation of AI systems. However, the leaders perceived that, for some healthcare professionals, these ideas are unthinkable, and it may take several years before these changes in roles and care processes become a reality in the healthcare sector.

I think I will be seeing other professions in the healthcare services who have perhaps not received a healthcare education. It will be a culture shock, I think. It also concerns that you may perhaps not need to be medically trained, for sitting there and checking those yellow flags or whatever they are, or it could perhaps be another type of professional group. I think that it would actually be good. We have to start economizing with the competencies we now have and it’s difficult enough to manage. Leader 15.

The acceptance of the AI systems may vary within and between professional groups, ages, and areas of specialized care. The leaders feared that the implementation of AI systems would change physicians’ knowledge base and that there would be a loss of knowledge that could be problematic in the long run. The leaders argued that younger, more recently graduated physicians would never be able to accumulate the experience-based knowledge to the extent that their older colleagues have done, as they will rely more on AI systems to support their decisions. Thus, on one hand, professional roles and self-images might be threatened when output from the AI systems is argued to be more valid than the recommendation by an experienced physician. However, on the other hand, physicians who do not “work with their hands” can utilize such output as decision support to complement their experience-based knowledge. Thus, it is important that healthcare professionals have trust in recommendations from the AI systems in clinical practice. If some healthcare professionals do not trust the AI systems and their output, there is a risk that they will not use them in clinical practice and continue to work in the way they are used to, resulting in two parallel systems. This might be problematic, both for the work environment and the healthcare professionals’ wellbeing. The leaders emphasized that this would represent a challenge for the implementation of AI systems in healthcare.

We can’t add anything more today without taking something else away, I’d say it was impossible. // The level of burden is so high today so it’s difficult to see, it’s not sufficient to say that this will be of use to us in two years’ time. Leader 20.

Implementing AI systems can change existing care processes and change the role of the patient. The leaders described that, in primary care, AI systems have the best potential to change existing work processes and make care more efficient, for example through an automatic AI-based triage for patients. The AI system could take the anamnesis, instead of the healthcare professionals, and do this when patients still are at home, so the healthcare professionals will not meet the patient unless the AI system has decided that it is necessary. The AI system can also autonomously discover something in a patient’s health status and suggest that the patient contact healthcare staff for follow-up. This use of AI systems could open up opportunities for more proactive and personalized care.

The leaders also described that the implementation of AI systems in practice could facilitate an altered patient role. The development that is taking place in the healthcare sector with, for instance, patient-reported data, enables and, in some cases, requires an active and committed patient that takes part in his or her care process. The leaders mentioned that there might be a need for patient support. Otherwise, there might be a risk that only patients with high digital literacy would be able to participate with valid data. The leaders described that AI systems could facilitate this development, by recommending self-care advice to patients or empowering them to make decisions. Still, there were concerns that not all patients would benefit from AI systems, due to variations in patients’ capabilities and literacy.

We also deal with people who are ill, we must also have respect for that. Everyone will not be able to use these tools. Leader 7.

Building trust for AI systems acceptance in clinical practice

A challenge and prerequisite for implementing AI systems in healthcare is that the technology meets expectations on quality to support the healthcare professionals in their practical work, such as having a solid evidence base, being thoroughly validated and meeting requirements for equality. It is important to have confidence in the validity of the data, the algorithms and their output. A key challenge pointed out was the need to have a sufficiently large population base, the “right” type of data and the right populations to build valid AI systems. For common conditions, where rich data exists to base AI algorithms, leaders believed the reliability would be high. For unusual conditions, there were concerns that there would be lower accuracy. Questions were also raised about how AI systems take aspects around equity and equality into account, such as gender and ethnicity. The leaders expressed concern that, due to these obstacles, in relation to certain unusual or complex conditions AI systems might not be suitable.

Then there is a challenge with the new technology, whether it’s Ok to apply it. Because it’s people who are affected, people’s health and lives that are affected by the new technology. How can we guarantee that it delivers what it says it will deliver? It must be safe and reviewed, validated and evidence-based in order for us to be able to use it. If a bug is built in then the consequences can be enormous. Leader 2.

Lack of confidence in the reliability of AI systems was also described and will place higher demands and requirements on their accuracy than on similar assessments made by humans. Thus, acceptance depends on confidence in AI systems as highly sensitive and that they can diagnose conditions at earlier stages than skilled healthcare professionals. The leaders perceived that the “black box” needs to be understood in order to be reliable, i.e. what the AI algorithms calculations are based on. Thus, reliance on the outputs from AI algorithms depends on reliance on the algorithm itself and the data used for its calculation.

There are a number of inherent problems with AI. It’s a little black box. AI looks at all the data. AI is not often easy to explain, “oh, you’ve got a risk, that it passed the cut-off value for that person or patient”, no because it weighs up perhaps a hundred different dimensions in a mathematical model. AI models are often called a black box and there have been many attempts at opening that box. The clinics are a bit skeptical then when they are not able to, they just get a risk score, I would say. Leader 10.

Big data sets are important for quality, but the leaders stated that too much information about a patient also could be problematic. There is a risk that information about a patient is available to healthcare professionals who should not have that information. The leaders believed that this could already be a problem today, but that it would be an increased risk in the future. This challenge needs to be handled as the amount of patient information increases, and as more healthcare professionals get access to such information when it’s being used in AI systems, regardless of the reason for the patient’s contact with the healthcare unit. Another challenge and prerequisite for implementing AI systems in healthcare is that the technology is user-friendly and create value for both healthcare professionals and patients. The leaders expected AI systems to be user-friendly, self-instructing, and easy to use, without requiring too much prior knowledge or training. In addition to being easy to use, the AI systems must also be time-saving and never time-consuming or dependent on the addition of yet more digital operative systems to work with. Using AI systems should, in some cases, be equated with having a second opinion from a colleague, when it comes to simplicity and time consumption.

An easy way to receive this support is needed. One needs to ask a number of questions in order to receive the correct information. But it mustn’t be too complicated, and it mustn’t take time, then nothing will come of it. Leader 4.

The leaders expected that AI systems would place the patients in focus and thereby contribute to more person-centred care. These expectations are based on a large amount of data on which AI algorithms are built, which leaders perceive will make it possible to individualize assessments and treatment options. AI systems would enable more person-centred and value-creating care for patients. AI systems could potentially contribute to making healthcare efficient without compromising quality. It was seen as an opportunity to meet future increasing needs for care among the citizens, combined with a reduced number of healthcare professionals. Smart and efficient AI systems used in investigations, assessments, and treatments can streamline care and allow more patients to receive care. Making healthcare efficient was also about the idea that AI systems should contribute to improved communication within and between caregivers for both public and private care. Using AI systems to follow up the given care and to evaluate the quality of care with other caregivers was highlighted, along with the risk that the increased efficiency provided by AI systems could result in a loss of essential values for healthcare and in impaired care.

I think that automatization via AI would be a safe way and it would be perfect for the primary care services. It would have entailed that we have more hands, that we can meet the patients who need to be met and that we can meet more often and for longer periods and perhaps do more house calls and just be there where we are needed a little more and help these a bit more easily. Leader 13.

The perspectives of the challenges described by leaders in the present study are an important contribution to improving knowledge regarding the determinants influencing the implementation of AI systems in healthcare. Our results showed that healthcare leaders perceived challenges to AI implementation concerning the handling of conditions external to the healthcare system, the building of internal capacity for strategic change management and the transformation of professional roles and practices. While implementation science has advanced the knowledge concerning determinants for successful implementation of digital technology in healthcare [ 53 ], our study is one of the few that have investigated leaders’ perceptions of the implementation of AI systems in healthcare. Our findings demonstrate that the leaders concerns do not lie so much with the specific technological nuances of AI, but with the more general factors relating to how such AI systems can be channeled into routine service organization, regulation and practice delivery. These findings demonstrate the breadth of concerns that leaders perceive are important for the successful application of AI systems and therefore suggest areas for further advancements in research and practice. However, the findings also demonstrate a potential risk that, even in a county council where there is a high level of investment and strategic support for AI systems, there is a lack of technical expertise and awareness of AI specific challenges that might be encountered. This could cause challenges to the collaboration between the developers of AI systems and healthcare leaders if there is a cognitive dissonance about the nature and scope of the problem they are seeking to address, and the practical and technical details of both AI systems and healthcare operational issues [ 7 ]. This suggests the need for people who are conversant in languages of both stakeholder groups maybe necessary to facilitate communication and collaboration across professional boundaries [ 54 ]. Importantly, these findings demonstrate that addressing the technological challenges of AI alone is unlikely to be sufficient to support their adoption into healthcare services, and AI developers are likely to need to collaborate with those with expertise in healthcare implementation and improvement scientists in order to address the wider systems issues that this study has identified.

The healthcare leaders perceived challenges resulting from external conditions and circumstances, such as ambiguities in existing laws and sharing data between organizations. The external conditions highlighted in our study resonate with the outer setting in the implementation framework CFIR [ 37 ], which is described in terms of governmental and other bodies that exercise control, with the help of policies and incentives that influence readiness to implement innovations in practice. These challenges described in our study resulted in uncertainties concerning responsibilities in relation to the development and implementation of AI systems and what one was allowed to do, giving rise to legal and ethical considerations. The external conditions and circumstances were recognized by the leaders as having considerable impact on the possibility of implementing AI systems in practice although they recognized that these were beyond their direct influence. This suggests that, when it comes to the implementation of AI systems, the influence of individual leaders is largely restricted and bounded. Healthcare leaders in our study perceived that policy and regulation cannot keep up with the national interest in implementing AI systems in healthcare. Here, concerted and unified national authority initiatives are required according to the leaders. Despite the fact that the introduction of AI systems in healthcare appears to be inevitable, the consideration of existing regulatory and ethical mechanisms appears to be slow [ 16 , 18 ]. Additionally, another challenge attributable to the setting was the lack of to increase the competence and expertise among professionals in AI systems, which could be a potential barrier to the implementation of AI in practice. The leaders reflected on the need for future higher education programs to provide healthcare professionals with better knowledge of AI systems and its use in practice. Although digital literacy is described as important for healthcare professionals [ 55 , 56 ], higher education faces many challenges in meeting emerging requirements and demands of society and healthcare.

The healthcare leaders addressed the fact that the healthcare system’s internal capacity for strategic change management is a hugh challenge, but at the same time of great importance for successful and sustainable implementation of AI systems in the county council. The leaders highlighted the need to create an infrastructure and joint venture, with common structures and processes for the promotion of the capability to work with implementation strategies of AI systems at a regional level. This was needed to obtain a lasting improvement throughout the organization and to meet organizational goals, objectives, and missions. Thus, this highlights that the implementation of change within an organization is a complex process that does not solely depend on individual healthcare professionals’ change responses [ 57 ]. We need to focus on factors such as organisational capacity, climate, culture and leadership, which are common factors within the “inner context” in CFIR [ 37 ]. The capacity to put the innovations into practice consists of activities related to maintaining a functioning organization and delivery system [ 58 ]. Implementation research has most often focused on implementation of various individual, evidence-based practices, typically (digitally) health interventions [ 59 ]. However, AI implementation represents a more substantial and more disruptive form of change than typically involved in implementing new practices in healthcare [ 60 ]. Although there are likely many similarities between AI systems and other new digital technologies implemented in healthcare, there may also be important differences. For example, our results and other AI research has acknowledged that the lack of transparency (i.e. the “black box” problem) might yield resistance to some AI systems [ 61 ]. This problem is probably less apparent when implementing various evidence-based practices based on empirical research conducted according to well-established principles to be trustworthy [ 62 ]. Ethical and trust issues were also highlighted in our study as playing a more prominent role in AI implementation, perhaps more prominently than in “traditional” implementation of evidence-based practices. There might thus be AI-specific characteristics that are not really part of existing frameworks and models currently used in implementation science.

Transformation of healthcare professions and healthcare practice

The healthcare leaders perceived that the use of AI in practice could transform professional roles and practices and this could be an implementation challenge. They reflected on how the implementation of AI systems would potentially impact provider-patient relationships and how the shifts in professional roles and responsibilities in the service system could potentially lead to changes in clinical processes of care. The leaders’ concerns related to the compatibility of new ways of working with existing practice, which is an important innovation characteristic highlighted in the Diffusion of Innovation theory [ 63 ]. According to the theory, compatibility with existing values and past experiences facilitates implementation. The leaders in our study also argued that it was important to see the value of AI systems for both professionals and service-users. Unless the benefits of using AI systems are observable healthcare professionals will be reluctant to drive the implementation forward. The importance of observability for adoption of innovations is also addressed in the Diffusion of Innovation theory [ 63 ], being the degree to which the results of an innovation are visible to the users. The leaders in our study conveyed the importance for healthcare professionals of having trust and confidence in the use of AI systems. They discussed uncertainties regarding accountability and liability in situations where AI systems impacts directly or indirectly on human healthcare, and how ambiguity and uncertainty about AI systems could lead to healthcare workers having a lack of trust in the technology. Trust in relation to AI systems is well reflected on as a challenge in research in healthcare [ 30 , 41 , 64 , 65 , 66 ]. The leaders also perceived that the expectations of patient-centeredness and usability (efficacy and usefulness) for service users could be a potential challenge in connection with AI implementation. Their concerns are echoed in a review by Buchanan et al. [ 67 ], in which it was observed that the use of AI systems could serve to weaken the person-centred relationships between healthcare professionals and patients.

In summary, the expectations for AI in healthcare are high in society and the technological impetus is strong. A lack of “translation” of the technology is in some ways part of the initial difficulties of implementing AI, because implementation strategies still need to be developed that might facilitate testing and clinical use of AI to demonstrate its value in regular healthcare practice. Our results relate well to the implementation science literature, identifying implementation challenges attributable to both external and internal conditions and circumstances [ 37 , 68 , 69 ] and the characteristics of the innovation [ 37 , 63 ]. However, the leaders in our study also pointed out the importance of establishing an infrastructure and common strategies for change management on the system level in healthcare. Thus, introducing AI systems and the required changes in healthcare practice should not only be dependent on early adopters at the particular units. This resonates with the Theory of Organizational Readiness for Change [ 70 ], which emphasizes the importance of an organization being both willing and able to implement an innovation [ 71 ]. The theory posits that, although organizational willingness is one of the factors that may facilitate the introduction of an innovation into practice, both the organization’s general capacities and its innovation-specific capacities for adoption and sustained use of an innovation are key to all phases in the implementation process [ 71 ].

Methodological considerations

In qualitative research, the concepts credibility, dependability, and transferability are used to describe different aspects of trustworthiness [ 72 ]. Credibility was strengthened by the purposeful sample of participants with various experiences and a crucial role in any implementation process. It is considered of great relevance to investigate the challenges that leaders in the county council expressed concerning the implementation of various AI systems in healthcare, albeit the preparation for implementing AI systems is a current issue in many Swedish county councils. Furthermore, the research team members’ familiarity with the methodology, together with their complementary knowledge and backgrounds enabled a more nuanced and profound, in-depth analysis of the empirical material and was another strength of the study.

Dependability was strengthened by using an interview guide to ensure that the same opening questions were put to all participants and that they were encouraged to talk openly. Because this study took place during the COVID-19 pandemic, the interviews were performed either at a distance, using the Microsoft Teams application, or face-to-face, the variation might be a limitation. However, according to Archibald et al. [ 73 ], distance interviewing with videoconferencing services, such as Microsoft Teams, could be beneficial and even preferred. Based on the knowledge gap regarding implementation of AI systems in healthcare, the authors chose to use an inductive qualitative approach to the exploration of healthcare leaders’ perceptions of implementation challenges. It might be that the implementation of AI systems largely aligns with the implementation of other digital technologies or techniques in healthcare. A strength of our study is that it focuses on perceptions on AI systems in general regardless of the type of AI algorithm or the context or area of application. However, one potential limitation of this approach is the possibility that more specific AI systems and or areas of applications may become associated with somewhat different challenges. Further studies specifying such boundaries will provide more specific answers but will probably also require the investigation be conducted in connection with the actual implementation of a specific AI systems and based on participants' experiences of having participated in the implementation process. With this in mind, we encourage future research to take this into account when deciding upon study designs.

Transferability was strengthened by a rich presentation of the results along with appropriate quotations. However, a limitation could be that all healthcare leaders work in the same county council, so transferability to other county councils must be considered with caution. In addition, an important contextual factor that might have an impact on whether, and how, the findings observed in this study will occur in other settings as well, concerns the nature of, and approach to, AI implementation. AI could be considered a rather broad concept, and while we adopted a broad and general approach to AI systems in order to understand healthcare leader’s perceptions, we would, perhaps, expect that more specific AI systems and or areas of applications become associated with different challenges. Taken together, these are aspects that may affect the possibilities for our results to be portable or transferred to other contexts. We thus suggest that the perceptions of healthcare leaders in other empirical contexts and the involvement of both more specific and broader AI systems are utilized in the study designs of future research.

In conclusion, the healthcare leaders highlighted several implementation challenges in relation to AI within the healthcare system and beyond the healthcare organization. The challenges comprised conditions external to the healthcare system, internal capacity for strategic change management, and transformation of healthcare professions and healthcare practice. Based on our findings, there is a need to see the implementation of AI system in healthcare as a changing learning process at all organizational levels, necessitating a healthcare system that applies more nuanced systems thinking. It is crucial to involve and collaborate with stakeholders and users inside the regional healthcare system itself and other actors outside the organization in order to succeed in developing and applying system thinking on implementation of AI. Given that the preparation for implementing AI systems is a current and shared issue in many (Swedish) county councils and other countries, and that our study is limited to one specific county council context, we encourage future studies in other contexts, in order to corroborate the findings.

Availability of data and materials

Empirical material generated and/or analyzed during the current study are not publicly available, but are available from the corresponding author on reasonable request.

Buch VH, Ahmed I, Maruthappu M. Artificial intelligence in medicine: current trends and future possibilities. Br J Gen Pract. 2018;68(668):143–4.

Article   PubMed   PubMed Central   Google Scholar  

Mehta N, Pandit A, Shukla S. Transforming healthcare with big data analytics and artificial intelligence: A systematic mapping study. J Biomed Inform. 2019;100: 103311.

Article   PubMed   Google Scholar  

Horgan D, Romao M, Morré SA, Kalra D. Artificial Intelligence: Power for Civilisation - and for Better Healthcare. Public Health Genomics. 2019;22(5–6):145–61.

Union E. A definition of AI: Main capabilities and scientific disciplines. https://ec.europa.eu/futurium/en/system/files/ged/ai_hleg_definition_of_ai_18_december_1.pdf ; 2018.

Davenport T, Kalakota R. The potential for artificial intelligence in healthcare. Future Healthc J. 2019;6(2):94–8.

Shaw J, Rudzicz F, Jamieson T, Goldfarb A. Artificial Intelligence and the Implementation Challenge. J Med Internet Res. 2019;21(7):e13659.

Elish MC. The Stakes of Uncertainty: Developing and Integrating Machine Learning in Clinical Care. Ethnographic Praxis in Industry Conference Proceedings. 2018;2018(1):364–80.

Article   Google Scholar  

Lee JC. The perils of artificial intelligence in healthcare: Disease diagnosis and treatment. J Comput Biol Bioinformatics Res. 2019;9(1):1–6.

Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25(1):44–56.

Article   CAS   PubMed   Google Scholar  

Technology EIoIa. Transforming healthcare with AI. https://eithealth.eu/wp-content/uploads/2020/03/EIT-Health-and-McKinsey_Transforming-Healthcare-with-AI.pdf ; 2020.

Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, et al. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol. 2017;2(4):230–43.

De Nigris S, Craglia M, Nepelski D, Hradec J, Gomez-Gonzales E, Gomez Gutierrez E, et al. AI Watch : AI Uptake in Health and Healthcare. 2020.

He J, Baxter SL, Xu J, Xu J, Zhou X, Zhang K. The practical implementation of artificial intelligence technologies in medicine. Nat Med. 2019;25(1):30–6.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Alhashmi S, Alshurideh M, Al Kurdi B, Salloum S. A Systematic Review of the Factors Affecting the Artificial Intelligence Implementation in the Health Care Sector. 2020. p. 37–49.

Google Scholar  

Asan O, Bayrak AE, Choudhury A. Artificial Intelligence and Human Trust in Healthcare: Focus on Clinicians. J Med Internet Res. 2020;22(6):e15154.

Gooding P, Kariotis T. Ethics and Law in Research on Algorithmic and Data-Driven Technology in Mental Health Care: Scoping Review. JMIR Ment Health. 2021;8(6):e24668.

Beil M, Proft I, van Heerden D, Sviri S, van Heerden PV. Ethical considerations about artificial intelligence for prognostication in intensive care. Intensive Care Med Exp. 2019;7(1):70.

Murphy K, Di Ruggiero E, Upshur R, Willison DJ, Malhotra N, Cai JC, et al. Artificial intelligence for good health: a scoping review of the ethics literature. BMC Med Ethics. 2021;22(1):14.

Choudhury A, Asan O. Role of Artificial Intelligence in Patient Safety Outcomes: Systematic Literature Review. JMIR Med Inform. 2020;8(7):e18599.

Fernandes M, Vieira SM, Leite F, Palos C, Finkelstein S, Sousa JMC. Clinical Decision Support Systems for Triage in the Emergency Department using Intelligent Systems: a Review. Artif Intell Med. 2020;102:101762.

Yin J, Ngiam KY, Teo HH. Role of Artificial Intelligence Applications in Real-Life Clinical Practice: Systematic Review. J Med Internet Res. 2021;23(4):e25759.

Wolff J, Pauling J, Keck A, Baumbach J. The Economic Impact of Artificial Intelligence in Health Care: Systematic Review. J Med Internet Res. 2020;22(2):e16866.

Gama F, Tyskbo D, Nygren J, Barlow J, Reed J, Svedberg P. Implementation Frameworks for Artificial Intelligence Translation Into Health Care Practice: Scoping Review. J Med Internet Res. 2022;24(1):e32215.

Safi S, Thiessen T, Schmailzl KJ. Acceptance and Resistance of New Digital Technologies in Medicine: Qualitative Study. JMIR Res Protoc. 2018;7(12):e11072.

Whitelaw S, Mamas MA, Topol E, Van Spall HGC. Applications of digital technology in COVID-19 pandemic planning and response. Lancet Digit Health. 2020;2(8):e435–40.

Alami H, Lehoux P, Denis J-L, Motulsky A, Petitgand C, Savoldelli M, et al. Organizational readiness for artificial intelligence in health care: insights for decision-making and practice. J Health Organ Manag. 2021;35(1):106–14.

Reed JE, Howe C, Doyle C, Bell D. Simple rules for evidence translation in complex systems: A qualitative study. BMC Med. 2018;16(1):92.

Reichenpfader U, Carlfjord S, Nilsen P. Leadership in evidence-based practice: a systematic review. Leadersh Health Serv (Bradf Engl). 2015;28(4):298–316.

Nilsen P, Bernhardsson S. Context matters in implementation science: a scoping review of determinant frameworks that describe contextual determinants for implementation outcomes. BMC Health Serv Res. 2019;19(1):189.

Chen M, Decary M. Artificial intelligence in healthcare: An essential guide for health leaders. Healthc Manage Forum. 2020;33(1):10–8.

Loh E. Medicine and the rise of the robots: a qualitative review of recent advances of artificial intelligence in health. BMJ Leader. 2018;2(2):59.

Ogbonna E, Harris LC. Leadership style, organizational culture and performance: empirical evidence from UK companies. Int J Human Res Manage. 2000;11(4):766–88.

Battilana J, Gilmartin M, Sengul M, Pache A-C, Alexander JA. Leadership competencies for implementing planned organizational change. Leadersh Q. 2010;21(3):422–38.

Denti L, Hemlin S. Leadership and innovation in organizations: a systematic review of factors that mediate or moderate the relationship. Int J Innov Manag. 2012;16(03):1240007.

Nilsen P. Overview of theories, models and frameworks in implementation science. In: Nilsen P, Birken, S. A., editor. Handbook on Implementation Science. Cheltenham: Edward Elgar Publishing Limited; 2020. p. 8–31.  https://www.elgaronline.com/view/edcoll/9781788975988/9781788975988.xml .

Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care. 1998;7(3):149–58.

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JCJIS. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. 2009;4.

Blase KA, Van Dyke M, Fixsen DL, Bailey FW. Key concepts, themes, and evidence for practitioners in educational psychology. In: Handbook of implementation science for psychology in education. New York, NY, US: Cambridge University Press; 2012. p. 13–34.

Chapter   Google Scholar  

Flottorp SA, Oxman AD, Krause J, Musila NR, Wensing M, Godycki-Cwirko M, et al. A checklist for identifying determinants of practice: A systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8(1):35.

Wangmo T, Lipps M, Kressig RW, Ienca M. Ethical concerns with the use of intelligent assistive technology: findings from a qualitative study with professional stakeholders. BMC Med Ethics. 2019;20(1):98.

Shinners L, Aggar C, Grace S, Smith S. Exploring healthcare professionals’ understanding and experiences of artificial intelligence technology use in the delivery of healthcare: An integrative review. Health Informatics J. 2020;26(2):1225–36.

Laï MC, Brian M, Mamzer MF. Perceptions of artificial intelligence in healthcare: findings from a qualitative survey study among actors in France. J Transl Med. 2020;18(1):14.

Diprose WK, Buist N, Hua N, Thurier Q, Shand G, Robinson R. Physician understanding, explainability, and trust in a hypothetical machine learning risk calculator. J Am Med Inform Assoc. 2020;27(4):592–600.

Nelson CA, Pérez-Chada LM, Creadore A, Li SJ, Lo K, Manjaly P, et al. Patient Perspectives on the Use of Artificial Intelligence for Skin Cancer Screening: A Qualitative Study. JAMA Dermatol. 2020;156(5):501–12.

Petitgand C, Motulsky A, Denis JL, Régis C. Investigating the Barriers to Physician Adoption of an Artificial Intelligence- Based Decision Support System in Emergency Care: An Interpretative Qualitative Study. Stud Health Technol Inform. 2020;270:1001–5.

PubMed   Google Scholar  

Graneheim UH, Lindgren BM, Lundman B. Methodological challenges in qualitative content analysis: A discussion paper. Nurse Educ Today. 2017;56:29–34.

Krippendorff K. Content analysis : an introduction to its methodology. Thousand Oaks, Calif: SAGE; 2013.

Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.

Ashfaq A, Lingman M, Sant’Anna A, Nowaczyk S. Readmission prediction using deep learning on electronic health records. J Biomed Inform. 2019;97:103256.

Ashfaq A, Lönn S, Nilsson H, Eriksson JA, Kwatra J, Yasin ZM, et al. Data Resource Profile: Regional healthcare information platform in Halland. Sweden Int J Epidemiol. 2020;49(3):738–9.

Wärnestål P, Nygren J. Building an experience framework for a digital peer support service for children surviving from cancer. New York, New York, USA: Proceedings of the 12th International Conference on Interaction Design and Children Association for Computing Machinery; 2013. p. 269–72.

Reed MS, Curzon R. Stakeholder mapping for the governance of biosecurity: a literature review. J Integr Environ Sci. 2015;12(1):15–38.

Vis C, Bührmann L, Riper H, Ossebaard HC. Health technology assessment frameworks for eHealth: A systematic review. Int J Technol Assess Health Care. 2020;36(3):204–16.

Rosen MA, DiazGranados D, Dietz AS, Benishek LE, Thompson D, Pronovost PJ, et al. Teamwork in healthcare: Key discoveries enabling safer, high-quality care. Am Psychol. 2018;73(4):433–50.

Booth RG, Strudwick G, McBride S, O’Connor S, Solano López AL. How the nursing profession should adapt for a digital future. BMJ. 2021;373:n1190.

Article   PubMed Central   Google Scholar  

Foadi N, Varghese J. Digital competence - A Key Competence for Todays and Future Physicians. J Eur CME. 2022;11(1):2015200.

Nilsen P, Schildmeijer K, Ericsson C, Seing I, Birken S. Implementation of change in health care in Sweden: a qualitative study of professionals’ change responses. Implement Sci. 2019;14(1):51.

Wandersman A, Duffy J, Flaspohler P, Noonan R, Lubell K, Stillman L, et al. Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. Am J Community Psychol. 2008;41(3–4):171–81.

Nilsen P, Birken S. Prologue. In: Nilsen P, Birken, S. A., editor. Handbook on Implementation Science. Cheltenham: Edward Elgar Publishing Limited; 2020. p. 1–6.  https://www.elgaronline.com/view/edcoll/9781788975988/9781788975988.xml .

Scott WR. Institutional change and healthcare organizations : from professional dominance to managed care. Chicago: University of Chicago Press; 2000.

von Eschenbach WJ. Transparency and the Black Box Problem: Why We Do Not Trust AI. Philosophy & Technology. 2021;34:1607+.

Li S-A, Jeffs L, Barwick M, Stevens B. Organizational contextual features that influence the implementation of evidence-based practices across healthcare settings: a systematic integrative review. Syst Rev. 2018;7(1):72.

Rogers EM. Diffusion of innovations. New York: Free Press; 1995.

Matheny ME, Whicher D, Thadaney IS. Artificial Intelligence in Health Care: A Report From the National Academy of Medicine. JAMA. 2020;323(6):509–10.

Moorman LP. Principles for Real-World Implementation of Bedside Predictive Analytics Monitoring. Appl Clin Inform. 2021;12(4):888–96.

Lee D, Yoon SN. Application of Artificial Intelligence-Based Technologies in the Healthcare Industry: Opportunities and Challenges. Int J Environ Res Public Health. 2021;18(1):271.

Buchanan C, Howitt ML, Wilson R, Booth RG, Risling T, Bamford M. Predicted Influences of Artificial Intelligence on the Domains of Nursing: Scoping Review. JMIR Nursing. 2020;3(1):e23939.

Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, et al. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2012;69(2):123–57.

Powell BJ, Fernandez ME, Williams NJ, Aarons GA, Beidas RS, Lewis CC, et al. Enhancing the Impact of Implementation Strategies in Healthcare: A Research Agenda. Front Public Health. 2019;7:3.

Weiner BJ. A theory of organizational readiness for change. Implement Sci. 2009;4(1):67.

Scaccia JP, Cook BS, Lamont A, Wandersman A, Castellow J, Katz J, et al. A practical implementation science heuristic for organizational readiness: R = MC(2). J Community Psychol. 2015;43(4):484–501.

Graneheim UH, Lundman B. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Educ Today. 2004;24(2):105–12.

Archibald MM, Ambagtsheer RC, Casey MG, Lawless M. Using Zoom Videoconferencing for Qualitative Data Collection: Perceptions and Experiences of Researchers and Participants. Int J Qual Methods. 2019;18:1609406919874596.

World Medical Association Declaration of Helsinki. ethical principles for medical research involving human subjects. JAMA. 2013;310(20):2191–4.

Article   CAS   Google Scholar  

Council SR. Good Research Practice: Swedish Research Council; 2017 [Available from: https://www.vr.se/download/18.5639980c162791bbfe697882/1555334908942/Good-Research-Practice_VR_2017.pdf .

Download references

Acknowledgements

The authors would like to thank the participants who contributed to this study with their experiences.

All authors belong to the Healthcare Improvement Research Group at Halmstad University, https://hh.se/english/research/our-research/research-at-the-school-of-health-and-welfare/healthcare-improvement.html

Open access funding provided by Halmstad University. The funders for this study are the Swedish Government Innovation Agency Vinnova (grant 2019–04526) and the Knowledge Foundation (grant 20200208 01H). The funders were not involved in any aspect of study design, collection, analysis, interpretation of data, or in the writing or publication process.

Author information

Authors and affiliations.

School of Health and Welfare, Halmstad University, Box 823, 301 18, Halmstad, Sweden

Lena Petersson, Ingrid Larsson, Jens M. Nygren, Per Nilsen, Margit Neher, Julie E. Reed, Daniel Tyskbo & Petra Svedberg

Department of Health, Medicine and Caring Sciences, Division of Public Health, Faculty of Health Sciences, Linköping University, Linköping, Sweden

Department of Rehabilitation, School of Health Sciences, Jönköping University, Jönköping, Sweden

Margit Neher

You can also search for this author in PubMed   Google Scholar

Contributions

LP, JMN, JR, DT and PS together identified the research question and designed the study. Applications for funding and coproduction agreements were put in place by PS and JMN. Data collection (the interviews) was carried out by LP and DT. Data analysis was performed by LP, IL, JMN, PN, MN and PS and then discussed with all authors. The manuscript was drafted by LP, IL, JMN, PN, MN and PS. JR and DT provided critical revision of the paper in terms of important intellectual content. All authors have read and approved the final submitted version.

Corresponding author

Correspondence to Lena Petersson .

Ethics declarations

Ethics approval and consent to participate.

The study conforms to the principles outlined in the Declaration of Helsinki (74) and was approved by the Swedish Ethical Review Authority (no. 2020–06246). The study fulfilled the requirements of Swedish research: information, consent, confidentiality, and safety of the participants and is guided by the ethical principles of: autonomy, beneficence, non-maleficence, and justice (75). The participants were first informed about the study by e-post and, at the same time, were asked if they wanted to participate in the study. If they agreed to participate, they were verbally informed at the beginning of the interview about the purpose and the structure of the study and that they could withdraw their consent to participate at any time. Participation was voluntary and the respondents were informed about the ethical considerations of confidentiality. Informed consent was obtained from all participants prior to the interview.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no potential conflicts of interest with respect to the research, authorship, and publication of this article.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Petersson, L., Larsson, I., Nygren, J.M. et al. Challenges to implementing artificial intelligence in healthcare: a qualitative interview study with healthcare leaders in Sweden. BMC Health Serv Res 22 , 850 (2022). https://doi.org/10.1186/s12913-022-08215-8

Download citation

Received : 04 April 2022

Accepted : 20 June 2022

Published : 01 July 2022

DOI : https://doi.org/10.1186/s12913-022-08215-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Artificial intelligence
• Digital transformation
• Implementation
• Healthcare leaders
• Organizational change
• Qualitative methods
• Stakeholders

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

Advertisement

Tackle implementation challenges in project-based learning: a survey study of PBL e-learning platforms

• Development Article
• Published: 28 March 2023
• Volume 71 , pages 1179–1207, ( 2023 )

Cite this article

• Nanxi Meng 1 ,
• Yan Dong   ORCID: orcid.org/0000-0003-1678-6370 2 , 3 ,
• Dorian Roehrs 1 &
• Lin Luan 2 , 4  

3 Citations

1 Altmetric

Explore all metrics

Project-based learning (PBL) has been identified as an effective pedagogy for instructors to help students to learn interdisciplinary knowledge, problem-solving skills, modes of thinking, and collaborative practices through solving problems in a real-world context. However, previous studies reported that instructors from K-12 to tertiary learning environments found it challenging to implement such a pedagogy for various reasons. The emergence of PBL E-learning platforms in the recent decade has attracted increasing interest in adoption and seems to provide a solution to tackle the difficulties in PBL implementation. Yet little is known about designing these platforms and how they facilitate the PBL learning process and management. In the current study, we conducted a multiple case survey study on 16 PBL learning platforms in English and Chinese, collected data on their features and functions, categorized them according to their services provided, and analyzed how they tackle the implementation challenges. Additionally, we identified four trends in PBL development as pedagogy, the skills, and competence required for teachers and students to successfully carry out PBL via e-learning platforms and provide suggestions to improve and refine the platform design for educational technologists and related stakeholders. The limitations of this study and the future research direction are included.

Similar content being viewed by others

Project-Based Learning in Higher Education

Effective Learning Through Project-Based Learning: Collaboration, Community, Design, and Technology

Project-Based Learning as a Model for the Normalized Use of Digital Technologies: Facilitating Practical, Meaningful, and Learner-Centered Online Teaching and Learning

Avoid common mistakes on your manuscript.

Introduction

Project-based learning (PBL) is a systematic and transformative pedagogy that advocates for students to gain knowledge and skills through working for an extended period to investigate and respond to an authentic, contextualized, complex question, problem, or challenge (Thomas, 2000 ). Students demonstrate their knowledge and skills by creating tangible or intangible artifacts and presenting them to real audience group(s). As a result, students develop deep content knowledge and skills like critical thinking, collaboration, creativity, and communication (Barron & Darling-Hammond, 2008 ). Studies have identified the positive impact PBL had on students, including increased design proficiency, improved confidence and willingness to approach challenges (Thomas, 2000 ), developed thinking skills (Anazifa & Djukri, 2017 ), and interdisciplinary competence (Brassler & Dettmers, 2017 ).

Despite the usefulness and effectiveness of PBL pedagogy, concerns were voiced against adopting PBL, as many teachers found it challenging to implement and manage the learning process (Aldabbus, 2018 ). Often, teachers were responsible for providing learning materials, keeping up with students’ personal and in-group collaboration progress, checking up on students’ learning performance, providing timely feedback, and supporting sufficient interactions among students, teachers, and content. Hence, a solution that assists teachers in managing the mentioned efforts was highly desired.

With the development of technologies in education and the digitalization of learning, adopting electronic learning (e-learning) platforms, especially in elementary and secondary schools, has seen a rapid evolution (Cavus et al., 2021 ). E-learning platforms have become an indispensable component of the education experience for students, teachers, and other stakeholders. Provides the technical infrastructure on which e-learning activities can take place. The various functions embedded in e-learning platforms help teachers complete teaching duties, administrative and management assignments, and practice the e-learning platforms and teaching method (Kassymova et al., 2020 ), like project-based learning.

Seeing the increasing popularity of PBL worldwide and the need to manage the learning process, the e-learning platforms specialized in supporting PBL emerged and have gained interest from teachers and educational institutes. PBL learning platforms like Project Pals developed by the U.S., Dreamdo Schools by Finland, Cura by Australia, and EPBL by China were good examples. Their user groups have been growing. However, though all claimed to be PBL e-learning platforms, they were distinctive in the rationale of design, features and functions, and the mechanism supporting PBL practice and development.

Since 2020, the Covid-19 pandemic has deeply affected the global economy and profoundly changed the form of education. Numerous schools shut down during the pandemic, and the uncertainty of face-to-face instruction accelerated the transition to online and digitalized learning, making e-learning platforms a core component in K-12 education. Prior to 2022, the top challenges of online learning for K-12 learners include inequity of technology and materials, lack of online teaching knowledge (Vinson & Caukin, 2021 ), unstable internet connection, and unfamiliar with the learning platforms and software (Zuo et al., 2021 ), among many others. Looking into the post-pandemic era, many schools and districts continue to use e-learning technologies to assist students to achieve learning success (Mann et al., 2021 ). On the other hand, the emergence of PBL e-learning platforms posts challenges for educators to choose the right platform for the targeted learners. In the current study, we propose surveying the available PBL platforms in both the English and Chinese worlds to explore how these platforms support PBL practice and development.

Literature review

Project-based learning (pbl).

PBL is a student-centered pedagogy and an overall approach to the design of the learning environment (Krajcik & Blumenfeld, 2006 ). There are five key features of the PBL learning environment and procedures (Krajcik et al., 2003 ). PBL starts from a driving question or a real-world problem to be solved. Then, students engage in authentic, contextualized inquiry to explore the driving question and learn to apply essential ideas in the discipline. Third, students, teachers, and community members engage collaboratively in activities to find solutions to the driving question, which mirrors the process of solving challenges in a real-world context. Fourth, learning technologies provide scaffolds for students to expand their ability and engage more profound in the inquiry process. Last, students create tangible products to address the driving question and share the artifacts publicly with the stakeholders. After decades of research, researchers have established four constructive principles of PBL (Krajcik & Blumenfeld, 2006 ): (1) contextualized learning. Learning in an authentic, real-world context allows students to see the value and meaning of the tasks and activities they perform and generalize better to a broader range of situations. (2) Active construction of knowledge. PBL allows students to actively construct their knowledge through participating in real-world activities similar to those that experts from different fields engage in, to solve problems and develop artifacts. (3) Social interactions. In PBL, students achieve learning goals through social interactions and the sharing of knowledge, which creates a community of learners. (4) Apply cognitive tools for scaffolding. Technological tools amplify and expand what students can learn. Specifically, cognitive tools can help students with data collecting, accessing and visualizing, collaboration, project planning, implementation, and learning output through various formats (multimedia, digital board, etc.).

Research has revealed the positive impact of PBL in improving the learning outcomes of students, including the affective outcomes (perceptions of the benefits and experience of PBL), cognitive outcomes (knowledge and cognitive strategies), behavioral outcomes (skills and engagement), and artifact outcomes (Guo et al., 2020 ). Despite the majority of research on PBL’s positive effects being conducted with students from higher education as participants (Verstegen et al., 2016 ), studies also identified positive learning results from K-12 learners worldwide (i.e., Kokotsaki et al., 2016 ). In terms of the learning experience, implementing PBL can improve the quality of learning for elementary students (Fauzia & Kelana, 2021 ), yield higher learning outcomes (Amini et al., 2019 ), and increase the affective connection (Virtue & Hinnant-Crawford, 2019 ). Implementing PBL improves students' academic achievement and knowledge retention (Al-Balushi & Al-Aamri, 2014 ), increasing the student's science process skills and cognitive learning outcomes (Nasir et al., 2019 ). Behaviorally, PBL improves students’ informational reading skills (Duke et al., 2021 ), social-emotional skills (Culclasure et al., 2019 ), collaboration and conflict-solving skills (Lee et al., 2015 ), problem-solving and critical thinking skills (Trisdiono et al., 2019 ). Studies with large effect sizes show the predominant benefits of PBL on K-12 learners.

The covid pandemic permanently changed learning behavior worldwide. Prior to 2022, the top challenges of online learning for K-12 learners include inequity of technology and materials, lack of online teaching knowledge (Vinson & Caukin, 2021 ), unstable internet connection, and unfamiliar with the learning platforms and software (Zuo et al., 2021 ), among many others. Yet looking into the post-pandemic time, many schools and districts used online learning to respond to the pandemic and continue to use e-learning technologies to assist students to achieve learning success (Mann et al., 2021 ). Owens and Hite's ( 2022 ) study reported implementing a STEM learning experience using a virtual global collaboration project-based learning approach using Canvas as the learning platform and achieved satisfying learning results. Given these, e-learning platforms and learning applications help teachers and students overcome the learning challenges and will continue to play important role in K-12 learning, especially in the PBL context.

Problems and challenges of PBL implementation

Despite the advantages of PBL mentioned above, implementing PBL in K-12 classrooms is documented as challenging. The challenges in K-12 classrooms are mainly four: beliefs and understanding of PBL, project design and plan, implementation management, and support. The challenges are listed in Table 1 .

From the numbers of challenges reported in Table 1 , project implementation and management were the top concerns of teachers, followed by project design and planning, support and beliefs, and understanding of PBL. Despite the challenges mentioned above, empirical evidence has demonstrated that PBL is a practical pedagogy and a must-have experience for students to enter the knowledge-based economy with 21st-century skills. Kokotsaki et al. ( 2016 ) listed five facilitating factors to smoothen the implementation of PBL instruction, including (1) adopting digital technology to engage students in designing and developing the project with guidance and support, (2) engaging students in collaboration and peer interaction with positive interdependence, individual accountability, equal participation, and social skills, (3) effectively scaffolding students’ learning, (4) providing support from administrators, (5) adopting two-phase PBL approach, for students to first acquire the sufficient competence by developing required knowledge and skills, before design and make products independently. Lewis et al. ( 2019 ) also proposed that educational technologists should be considered to aid, scoping, curriculum, and coordinating tools to facilitate the implementation of PBL.

Identifying the facilitating factors is only the first step, without properly solving the challenges can hinder teachers and their passion for adopting PBL continuously in their classrooms. The challenges instructors perceive in the classroom practices should fuel educational technologists to overcome these challenges and communicate the benefits (Rogers, 2003 ). Given the rapid development of educational technologies, it is promising to seek solutions to the abovementioned issues and put the proposed strategies into practice.

E-learning and E-learning platforms

With the development of technology and its impact on education, it is almost impossible to teach and learn without the support of ICTs today. E-learning refers to using ICTs to facilitate and support learning (JISC, 2014 ). ICTs here could be applications, programs, objects, websites, etc., as long as it provides learning opportunities for individuals (Moore et al., 2011 ). The e-learning platforms could be e-learning systems, learning management systems (LMS), course management systems (CMS), virtual learning environments (VLE), or other websites and mobile applications that support learning. After decades of development, e-learning platforms are equipped with versatile functions, they provide a range of tools and facilities to help the interactive learning process. For instance, students and teachers can upload and get access to learning materials in a great variety of formats, interact with each other through communication tools like the message, forums, chats, and videoconferences, collaborate with peers, support assessment and reflection, and many more (Choudhury & Pattnaik, 2020 ; Donkers, Verstegen, de Leng, & de Jong, 2010). For students, e-learning increases the accessibility of learning (Dziuban et al., 2018 ). It allows students to have more control over the learning process (Blount, 2016 ) hence improving knowledge retention and learning performance (CITE), making learning flexible (Lara et al., 2014 ) and cost-effective (Farhan et al., 2018 ). For teachers, e-learning makes it easy to reuse, update and arrange course materials (Blount, 2016 ), and allows teachers to track learning data generated by students’ learning behaviors. These advantages of e-learning have convinced educational institutes worldwide to adopt it as part of the learning service for students (Toth-Stub, 2020).

In PBL implementation, adopting educational and technological tools is an essential practice that differentiates PBL from other pedagogies (Krajcik & Blumenfeld, 2006 ). Kokotsaki et al. ( 2016 ) listed technological tools as the primary enabler for students to engage smoothly with the PBL process. PBL practitioners and researchers have integrated various tools to facilitate teaching and learning in K-12 classrooms. Two tools are applied: individual tools with specific functions and existing e-learning systems that can partially support the PBL process. Individual tools used in PBL include contextual information providers like Google, Bing, and YouTube (Iwamoto et al., 2016 ), performance assessment tools like Performance Assessment Resource Bank (PARB, Guha et al., 2018 ), ICT tools for facilitating communication (Habók & Nagy, 2016 ), and collaboration tool like mentioned in the study of Rongbutsri ( 2017 ). E-learning systems applied as PBL management and facilitation tools include but do not exclude Google Classroom (Ramadhani et al., 2019 ) and Moodle (Wu & Wu, 2020 ). These systems are mainly applied to deliver information, facilitate communication and collaboration, and conduct assessments (Alverson et al., 2019 ). Despite the versatility of the embedded functions provided in the existing systems, instructors typically need to integrate additional tools to support PBL implementation, which increases the cognitive load for students and hampers the optimization of the learning outcome.

In recent years, a noticeable trend has been the emergence of e-learning systems specialized in supporting PBL worldwide. Platforms like ProjectPals from the US, Dreamdo School from Finland, and EPBL from China are attracting more and more K-12 users. The COVID-19 global pandemic has served as the catalyst and accelerated the progress of the transformation of educational institutes (Adnan & Anwar, 2020 ). PBL learning platforms are embracing their rapidly growing clientele. Yet little is known about these platforms, especially their rationale of design, features, functions, and how they tackle the implementation challenges and support PBL.

The study on PBL e-learning platforms is still in its infancy. The heterogeneity in instructional design and other features of the platforms are rarely explored and understood with empirical evidence. In the current study, we intend to conduct an exhaustive search in both the English and Chinese worlds on PBL e-learning platforms and seek to understand their innovative mechanism of supporting PBL. The research questions of this study are:

What are the properties of the selected PBL e-learning platforms, and what functions do they possess?

How do the platforms tackle the PBL implementation challenges?

How is PBL as a pedagogy understood differently through the design of the platforms?

The aim of the study is not to prove the usability or efficiency evaluation of these platforms but to uncover how PBL as pedagogy is understood differently through the design of the platforms, how these platforms solve challenges of the PBL practice, and how they facilitate PBL in different learning modes and environment. With the gained insights, we propose to holistically understand the wisdom of supporting PBL worldwide, provide guidance and recommendation for teachers, students, schools, and districts to select appropriate platforms, and shed light on how educational technologists could optimize the design of these platforms.

Methodology

In the current study, we adopted the multiple case study approach to explore the selected PBL platforms. Compared to single case studies, a multiple case study design allows researchers to develop a more in-depth understanding of the phenomena (Stake, 2013 ). This research design was appropriate for this study due to the nature of the research questions. Each selected PBL platform is an individual case, studied independently before drawing significant comparisons across cases.

The selection of the online PBL learning platforms

To answer the research questions, the researchers first conducted a platform search. The search was carried out both in English and in Chinese. All the searches were completed before September 2021. In the Google search engine, researchers first input the terms “project-based learning platform” and "global project-based learning platform" in English. There were 11 and five (5) results claimed to be PBL platforms. Besides google searching, researchers also identified one (1) platform mentioned in the educational technology review blogs. Excluding the overlapped results, there were 13 PBL platforms in English. The Chinese search was conducted by searching the same term with the Google search engine in both simplified and traditional Chinese and received three (3) results in simplified Chinese and one (1) result in traditional Chinese. The search result in traditional Chinese overlapped with one of the results in English.

There are four criteria we applied when selecting platforms from the results. Specifically, we eliminated: (1) platforms that support multiple pedagogies instead of focusing mainly on PBL; (2) platforms that provide only one or several functions associated with PBL (collaboration support, learning portfolio, etc.); (3) platforms only support the learning of one subject (i.e., language or programming); and (4) new PBL learning platforms with no actual users. In total, 16 (13 in English and 3 in Chinese) platforms were included in this study, they 13 platforms in English are: ProjectPals (PP, h www.projectpals.com/ ), Foundry (FD, https://www.projectfoundry.com/ ), Defined Learning (DL, https://www.definedlearning.com/ ), Educurious (EDC, https://educurious.org/ ), LiftLearning (LL, https://liftlearning.com/ ), Headrush Learning (HR, http://www.headrushlearning.com/ ), Sprocket (SK, https://sprocket.lucasedresearch.org/ ), Echo ( https://newtechnetwork.org/echo/ ), PBL Works (PBLW, https://www.pblworks.org/ ), Dreamdo Schools (DDS, https://edu.dream.do/en ), Cura (CR, https://www.curaeducation.com/ ), iEARN ( https://www.iearn.org/ ), and PenPal Schools ( https://www.penpalschools.com/ ). The three platforms in Chinese are: EPBL (EPBL, http://epbl.aicfe.cn/epbl/ ), GoPBL (GPBL, https://www.gopbl.com/ ), Creative Knowledge PBL Platform (CK, http://xms.forclass.net/ ). The information on all the platforms, including their full name, URL, country of origin, initiator and property, are briefly introduced in Table 2 in Appendix 1.

Data collection and validation

To guarantee the trustworthiness and validity of the collected data, we followed the data triangulation procedures and collected data from more than one source using more than one method (Connaway & Radford, 2016 ). The qualitative data collected for this study were from three different resources in different ways: (1) researchers’ observational journals and notes from visiting the website of each platform, (2) researchers’ user experience journal logged from using the platforms with the provided free trial account, and (3) the interactions researchers had with platform staff via the scheduled demonstration and follow-up emails. All the data included was collected before October 2021. Any updated or new functions added to the studied platforms were not included.

When visiting each platform’s website, the researchers first captured all the text presented for the coding later. The observational journals and notes for each platform were collected from two separate researchers before consolidating into one excel sheet for analysis. When using the free trial account on each platform, the researchers went through the entire platform to familiarize each function and adopted the think-out-loud method to keep the usability notes, which were transcribed and coded later for analysis.

When preparing the collected data for coding, the English discourse data were transcribed and coded with NVivo 12. The Chinese discourse data was transcribed with iflyrec.com and coded in an excel sheet in Chinese, and the results of the data analysis were conveyed in English to guarantee accuracy. All the quotes translated from Chinese into English were sent back to each participant to verify their correctness, especially those words and phrases that could not be directly translated into English. We followed Tie, Birks, & Francis (2019)’s guidance. Two researchers in the research team first completed the initial coding, did the constant comparative analysis, and established cross-case analysis.

We organized the coded data from three sources into tables (Table 2 in Appendix 1 and Table 3 in Appendix 2) to summarize the standard features within the same type of platforms and distinguish the differences across to answer the research questions. We first present the findings from the following aspects: properties, unique features and functions, problems and challenges solved, and rationale of the design.

General properties

The name of each platform, the coded name, and their URL are listed in Table 2 in Appendix 1. The 16 platforms came from five countries: The United States (10), China (3), Spain (1), Finland (1), and Australia (1). The targeted users were mainly K-12 educators and learners, except three platforms indicate supporting post-secondary users. All the platforms were web-based or websites, two claimed to have mobile applications, yet researchers could not identify them in the primary application stores. 13 platforms offer free-to-try or partially free-to-use accounts for individual users. Through interacting with the platform technologists, we identified four types of platform developers: entrepreneurs with K-12 classroom teaching or counseling experience, researchers and developers of higher educational institutes, global or domestic non-profit organizations, and educational corporations.

Special features and functions

Nine themes related to platform features and functions emerged from coding. We organized them into nine categories of functions: (1) project planning and building, (2) project management, (3) competency, (4) rubric, assessment, and feedback, (5) evidence and products, (6) learning analytics, (7) teacher professional development, (8) PBL community and ecosystem, and (9) learning mode and tech support. Table 3 shows functions embedded in each platform to the best of the researchers’ knowledge. Although all claimed to be PBL platforms, their services are diversified. We conducted a cross-case analysis and identified four types of PBL platforms: learning management platforms, content providers, communication and community facilitator, and service provider, as shown in Fig.  1 . It is worth noting that one platform can provide multiple types of services. Hence, we focused on their featured service(s) when categorizing them. We elaborate on the features of each type below.

Four Types of PBL Platforms

Learning management platforms

According to our study, learning management platforms account for 62.5% of the total studied platforms. There were ten platforms designed to manage the learning process of PBL. They were PP, FD, HR, LL, Echo, DL, DDS, EPBL, GPBL, and CK. This result matched our literature review that project management was the top concern of practitioners. Multiple technologists from the above platforms mentioned the rationale for them to create their products include: (1) the difficulty of managing the learning process, (2) the importance for teachers and students to acquire project management skills, and (3) the authenticity of a learning experience as the primary reasons for designing learning management platforms. As the designer from PP stated:

……. Companies are using software like Trello, Monday, and Jira to communicate not just within companies, countrywide and worldwide…… Project management in 2025 will become an 8 billion business, but we don't see much of that in education……teachers are not project managers, a big part of the success in PBL is to know how to manage projects……Students are expected to work as team members on projects when they enter the workplace. They should learn in the same way……

Similarly, the designer of GPBL also voiced the urgency of developing a management platform to guarantee the successful implementation of PBL, as he stated:

A learning project that allows students to display their learning agency requires teamwork, personalized learning tasks, timely assessment, and feedback, but facing a class with more than 30 students? It is an impossible mission for the teacher. There has to be a platform to support teachers if we are serious about implementing PBL.

Learning management platforms covered most functions across all nine categories, especially in project planning and designing, project management, interaction and communication, assessment and feedback, documentation and product, learning analytics, and teacher professional development. These platforms guided users through similar learning phases during the projects (terms varied), including project launch (discover, understand, plan, design), investigate (collaborate, manage), manage (monitor, track, capture), and share (report, adapt, reflect, reflect, thrive). Due to the various functions and the relatively long learning process, most platforms provide teacher training and familiarize teachers with the platform navigation. Some platforms (i.e., PP and GPBL) even make teacher training compulsory to reduce the anxiety and frustration that might be caused by cognitive overwhelming.

Depending on the service-providing mode and target users, the learning management platforms could be further divided into two subtypes: individual users and institute users. Platforms for individual users are PP, DL, HR, DDS, EPBL, GPBL, and CK. Although the trial use was issued only upon request, these platforms not only gave a single license for individual users to register as teachers or students but also supported institutional users in implementing learning projects. The flexibility of these platforms allows digitalized PBL to be implemented in a diversified learning context. As the technologist from DL stated:

……We also issue a single license to teachers of summer camps. All teachers are welcome to use our platform, for short or long. Our platform is flexible, as long as teachers want to try it…

The representative management platforms for institute users are FD, LL, and Echo. They offer customized services to assist the institute-wide transformation to PBL and fulfill the personalized needs of schools and districts. These three platforms did not provide a free trial to account; therefore, their functions marked in Table 3 only highlight their expertise.

The infrastructure in the classroom is an essential factor in how the project learning process is supported, which clustered the learning management platforms into (1) evidence-based documentary platforms and (2) process-oriented platforms. GPBL and CK are two good examples of evidence-based learning documentary platforms. They prioritize supporting the implementation of assessment plans, establishing learning portfolios by allowing uploading learning evidence of different formats throughout the learning process, and digitally showcasing the learning outcomes. GPBL and CK were both from China. According to the technologists, their platforms were evidence-based for two reasons. First, computers and other smart mobile devices were generally not provided beyond information technology classes. Students could not get constant access to devices during the project learning. Second, due to the tradition of entrance examination in middle and high schools, PBL is typically implemented in elementary education before students heavily engage in academic learning. Elementary school students had limited digital competency to carry out project learning and sharing purely online. Therefore, students typically use the platform at the end of each learning session to collect and store the fine pieces of digital learning evidence to form a learning portfolio and provide continuous tracking of their project learning experience.

Platforms like PP and HR are the representatives of process-oriented platforms. Both are designed for classrooms in which computers are accessible for students throughout the projects. Beyond learning evidence documentation, process-oriented platforms intended to bring the authentic experience of how projects are completed in real-world workplaces by allowing teachers and students to create projects from scratch, with various scaffolding tools embedded (i.e., visual organizers, analysis tools, event templates, etc.). By collaboratively drafting and editing the project pages and leaving comments on each created event, students can enhance their collaborative learning, problem-solving and critical thinking skills. The constructive learning process facilitates students’ learning agency. Additionally, allowing teachers to check personal and team learning progress can help track students’ performance and contribution in a group.

The anticipated learning environment is an essential factor. The technologists from the ten platforms unitedly stated that PBL should ideally implement in a traditional face-to-face classroom setting in K-12 education for a better experience of contextualized learning and close collaboration. It explained why platform that supports synchronous learning (i.e., videoconference) were rarely observed.

Although focusing on learning management, the ten LMPs were distinct from the traditional LMSs. In addition to serving the information delivering function, LMPs also embedded a number of pre-designed projects to help teachers familiarize themselves with the navigation and prepare them to customize and create their projects. DL and EPBL were content-intensive. Both platforms provided hundreds of pre-designed projects and scaffolding activities. The designers from both platforms emphasized the importance of supporting teachers by providing ready-to-use materials and their intention to establish a curriculum-based learning projects warehouse soon.

The LMPs were designed to solve a series of challenges in project planning, implementation, and support by providing the abovementioned functions. For students, the learning management platforms helped them to: (1) become familiar with the PBL learning process by providing the learning menu, (2) to keep up with the learning goal of the projects by referring to the embedded learning goal and project rubrics, and (3) supporting team collaboration by providing team learning space and data on learning progress. For teachers, the LMPs (1) integrated all the resources, materials, and information demanded by a project without leaving out the detail, (2) allowed teachers to monitor the learning process and keep track of individual/team learning progress, (3) provided sufficient a/synchronous assistance to help individual students and teams, and (4) empowered teachers to design and carry out authentic, multi-dimensional assessment, with referring to the different types of learning data. For external project facilitators, the platforms allowed them to join via invitation. They could contribute to enhancing the support of external experts and promoting school-parent collaboration, ultimately building the PBL learning ecology.

Content provider platforms (CPP)

Three platforms were categorized as CPP: EDC, SPK, and CR. EDC and SPK listed teachers as the target users and provided strong examples of PBL curricula and flexible teaching resources for students in grades 3–12 on various subjects. SPK materials were under a Creative Commons license as open educational resources (OER). Both platforms provided detailed project teaching materials on individual subjects or areas of study (i.e., biology, English, science, and social studies). EDC provided PBL courses designed jointly with local employers Footnote 1 to promote career-connected learning, which increased the authenticity of the learning projects. However, the project content on SPK was OERs created by teachers who joined the community of practice.

CR took a different route and provided pre-designed, self-paced PBL courses and learning modules directly to students. The technologist of CR defined it as a content platform. By delivering the asynchronous learning materials and scaffolding templates (i.e., team contract template, debate preparation form), CR intends to provide a PBL learning experience with the two-phase learning approach. In phase one, students should be familiar with the knowledge and skills related to the project's theme and prepare for the collaborative and creative work in phase two.

The content from the CPPs aligns with the pre-existing curriculum in their respective regions and countries, which contributed to solving two challenges: the mismatch of curriculum and PBL learning goals and practices and the difficulty of choosing and contextualizing important content.

Community organizer and communication facilitator platforms

Although platforms EDC, SPK, DDS, iEARN, and PPS branded themselves as PBL learning platforms, they specialize in creating professional learning and teaching communities and facilitate communication among students, teachers, and external facilitators. In the previous subsection, we listed EDC and SPK as the CPP, but both platforms also showed strong domestic social attributes. SPK created an online community and communication tools for teachers to share their experience adopting the provided materials and implementation insights. SPK also encourages teachers to collectively enact the curriculum materials and update and enhance the project-based programs to meet the needs of their local contexts. EDC provided an expert network by recruiting experts from diverse fields to contextualize their learning and help students tackle the challenges. DDS, iEARN, and PPS facilitated connecting teachers and students globally to communicate and collaborate on various projects via messages, discussion forums, and videoconference tools. Specifically, projects in iEARN aligned with the sustainable development goals of the United Nations and promote the endeavor of improving the quality of life worldwide.

Many LMPs support external teachers and facilitators collaboratively in designing, implementing, monitoring, and assessing the learning process. But the collaboration heavily relies on teachers' personal networking and connections. One of the significant advantages of the organizer and facilitator platforms like EDC, SPK, and DDS was that they created a community for like-minded teachers and students to experience authentic, cross-cultural collaboration in action. The platforms facilitating communication helped solve teachers' lack of pedagogical support and co-teaching opportunities, creating opportunities for students to learn projects cross-contextually and cross-culturally.

Service provider

PWS was the only platform we categorized as a PBL service provider, as it provides neither support directly to classroom learning nor curriculum to teachers, but professional development training and materials like videos, planning forms, rubrics, and blogs to teachers, to deepen the understanding of PBL as a pedagogy. Although only a limited number of projects were provided, these projects were designed to walk teachers through each step of project design following the project design rubrics. Unlike SPK provided discussion forums to group teachers to different topics and grade levels, PWS recruited faculty nationwide to create content and professional development materials on various world-concerned topics to tackle the frequently encountered challenges. The service of PWS went beyond supporting PBL practice, and it played the driving force behind the development of PBL pedagogy and its related research.

Understanding of PBL as a pedagogy

Through cross-case analysis, we observed that PBL as a pedagogy was understood differently through the design of the platforms. Specifically, the differences focused on four facets: (1) single subject vs. interdisciplinary learning projects, (2) teacher-led vs. student-led learning, (3) career-connected learning and the authenticity of PBL, and (4) PBL as a pedagogy for fostering global competence.

Single subject vs. interdisciplinary learning projects

PBL provides students opportunities for real-world challenges and questions, which are often interdisciplinary by nature (Repko et al., 2019 ). Among the studied platforms, we observed a divided understanding between PBL as a single subject and an interdisciplinary approach. On teacher-oriented platforms like SPK and all three platforms from China (EPBL, GPBL, and CK), projects were organized by subject (i.e., math, physics, biology). In contrast, platforms supporting students' learning process (i.e., PP, HR, DL) projects organized by areas of subjects showed more interdisciplinary nature. Besides the cultural differences in education, the emerging platforms providing interdisciplinary projects may act as a promoting force for implementing interdisciplinary learning in K-12.

Teacher-led vs. student-led PBL

With the collaborative and management features, many LMPs were designed to provide students more opportunities to develop the ownership of the projects and experience and increase students' learning agency by enabling personalized learning task assignments and collaboration. Yet the design of LMPs and their product discourse divided them into teacher-led (i.e., DL) and student-led (i.e., PP). DL was designed for teachers to lead student-centered learning. The teacher was the authorized party to cherry-pick and assign learning tasks from various materials according to the goal and objectives. In comparison, PP provided scaffolding tools and templates for students to establish a project from scratch or for teachers and students to co-create and co-plan. Students play the dominant role in their learning.

Career connected learning and the authenticity of PBL

Beyond supporting PBL, we observed platforms DL and EDC also featured for their career-connected learning (CCL). On DL, a similar product named Defined Careers™ provided a personalized career assessment and hands-on project learning for students to explore all career paths. On EDC, career-connected learning was a separate course unit that offered opportunities for high school students to participate in planning their future actively. Students could advocate themselves to employers and recruiters by completing the learning units.

The authenticity feature of PBL requires students to do work that is real to them, or the work directly impacts or uses in the real world. The goal of CCL is to connect learning to the real world, allow students to understand academic content in a way that is relevant to them, and help them develop knowledge, skills, and experiences to help them enter the world after school (Meeder & Pawlowski, 2020 ). Exploring the career path and being career ready are highly relevant to each student. Integrating CCL into PBL with academic content could be a viable path to preparing students to succeed in the global economy. Besides the currently available approaches mentioned in the work of Meeder and Pawlowski ( 2020 ), PBL platforms with CCL integrated may have the potential to support authentic CCL activities systematically.

PBL for global competence

Platform DDS, iRN, and PPS were all devoted to connecting students and classrooms worldwide, and they all had many users. Globally concerned issues were designed to be learning projects, and students from different countries and cultural backgrounds learned to compare, contrast, collaborate and contribute to solving the problems. Although the learning tasks on these platforms were not strictly following the typical learning phases, and students' collaboration was primarily through videoconferencing and writing, these platforms showed an innovative function of PBL: fostering global competence. With the themed communication promoted by these platforms, we also observed the trend that PBL broke the wall of individual classrooms and enhanced the development of cross-cultural competence.

To sum up, we studied 16 PBL platforms designed in English and Chinese-speaking regions, identified nine clusters of functions, and categorized the platforms into four types. Project management was the top challenge in PBL practice. We identified platforms that tackled the challenges by providing services to different user groups, learning environments, and learning modes. We also identified platforms specialized in delivering the content, facilitating the forming of learning and teaching communities, enabling communication beyond the classroom, and enhancing the understanding of PBL as a pedagogy. Reflected from the design of the platforms, PBL applied as a student-led or teacher-led pedagogy, an approach to develop career readiness and foster global and cross-cultural competence.

Discussion and implication

In this section, we discuss four observations based on studying PBL platforms and the skills and competence required for teachers and students to implement PBL successfully with the researched platforms.

Traditional LMS and LMPs

PBL LMPs, as a subcategory of LMS, were designed to provide for the unique needs of PBL. They offered not only the learning, communication support, and productivity functions but also the flexibility to accommodate the needs of K-12 teachers and learners within and beyond the learning projects. LMPs were different from the traditional LMS in at least three facets. First, LMPs went beyond information delivery and learning management functions (Kraleva et al., 2019 ) and provided teachers with both teaching materials and tools to facilitate the PBL design, implementation, teacher professional development, and advanced the development of PBL as a pedagogy. Second, LMPs realized personalized and collaborative learning through team member management tools, personalized learning task assignments, and learning progress tracking, which emphasized both the personalized learning experience and the collective effort to the overall success of the projects. Third, through co-plan and co-creation of learning projects, LMPs promoted students’ ownership through the learning process and participated in the projects with equal identity compared to the traditional LMS.

PBL platforms as warehouses and student autonomy

As mentioned previously, multiple platforms provided ample project examples and intended to build learning projects, activities, and tasks warehouses. Platforms (i.e., DL) even provide multiple investigation routes and formative assessment plans for students and teachers, which seemed to be a feasible way to encourage more teachers to adopt PBL. Studies also showed that teachers who felt well supported were more motivated to implement and persist in using PBL (Lam et al., 2010 ). Designing PBL platforms as warehouses may solve the difficulty of teacher project design and planning, lowering the challenge for students to raise driving questions and lead the investigation. Yet, it might compromise the opportunities to develop student autonomy.

Student autonomy was identified as one of the five essential characteristics of learning projects, and authentic PBL projects “do not end up at a predetermined outcome or take predetermined paths” (Thomas, 2000 , p. 4). Previous studies also identified that student choice and autonomy throughout the PBL process were helpful for students to develop a sense of ownership and control over their learning (Kokotsaki et al., 2016 ). How to balance and moderate the support for teachers to incorporate student autonomy, choice, unsupervised work time, and responsibility according to their competence should be considered for future platform design.

The trend of global classroom connection

The emergence of PBL communication facilitating platforms signaled the trend of globalization of education in K-12 and the importance of fostering students’ global competence. Unlike traditional LMS organizing learning mainly institute-wide or nationwide, PBL communication facilitating platforms built the globally connected classroom. Through the experience of studying the same projects and exchanging cultural and contextual information, students from different countries can collaborate synchronously and asynchronously through the discussion board, emails, and chats to collaborate and contribute to the globally concerned issues. This learning experience can improve their global digital citizenship, significantly impacting youth over the next decades (Harris & Johns, 2021 ). With the restricted international traveling during COVID and online and blended learning as the new normal in the post-COVID era, LMPs and communication facilitation platforms will play an essential role in fostering global competence. Various technologies that could bring students immersive experiences like augmented reality (AR) and virtual reality (VR) should integrate to simulate the hands-on international collaborative learning experience.

The proposed new skills

To fully appreciate the convenience of the PBL platforms, new skills are required for students and teachers: digital competence for students, TPACK knowledge and pedagogical digital competence for teachers, and project management skills for both.

To get teachers ready to effectively adopt PBL platforms, acquiring sound TPACK knowledge is a must. TPACK is featured to enable teachers to make intelligent pedagogical uses of technology (Koehler et al., 2007 ). It encompasses teachers’ expertise in technology integration, and focuses on teachers’ capacity of designing content, pedagogy, and knowledge of technology at all levels. When adopting the PBL platform, it is crucial for teachers to possess not only the relevant content and pedagogical knowledge but the technological and pedagogical technological knowledge (Koehler & Mishra, 2009 ). Specifically, teachers should actively keep up with the evolution of the development of the available e-learning platforms, think about how teaching and learning would change when platforms are used to support certain learning activities, and apply them productively to assist students to achieve their learning goals and help them internalize their skills of information technology adaption (Harris, 2016 ).

Pedagogical digital competence (From, 2017 ) requires the teacher to consider the interrelationships between knowledge, skills, attitudes, technology, learning theory, subject, context, and learning. With a thorough understanding, teachers could consistently plan, conduct, evaluate and revise ongoing teaching practice with theory, current research, and proven experience in a technology-supported teaching context to best support students learning. The innovative nature of PBL and PBL platforms requires teachers to improve their information and data literacy, digital communication and collaboration skills, create content digitally, problem shooting for teaching on the platforms, and innovatively implement PBL and develop it as a pedagogy.

Project management skills are another critical component for successfully implementing PBL. In the current study, we found that despite the differences, the PBL platforms, especially the learning management platforms, share the similar learning and teaching processes to a certain extent and can be break into three learning phases: (1) project planning, key words include discover, connect, define goals, plan, (co-)design, adopt and adjustment, (co-)create, and build, (2) launching and implementing, key words include launch, collaborate, monitor, respond, transform, track, capture, report, support, communicate, manage, assess, and (3) reporting and reflection, key aspects include report, reflect, adapt, publish, application, demonstration, showcase, evidence collection. These key words overlap with the skillsets required for successful project management in the real world.

Project management skills are essential for professions in organizations with upper management and multiple teams and departments involved in several extensive projects, like engineering, manufacturing, and construction (Cleland, 2007 ). PBL is featured for providing an authentic learning experience, which demands two types of project management skills for teachers: simulate the project management in the real-world context to demonstrate good project management skills for students, and help students to become good project managers by assisting them in organizing groups and monitoring their progress, improving their technical management skills like planning and forecasting, tracking and monitoring progress, enhance their subject matter expertise, and foster the soft skills like time management, leadership, and adaptability (Meredith et al., 2017 ). These will help maximize the benefit of adopting PBL pedagogy.

Conclusion, limitation, and future research

This article presented a multiple case study of 16 PBL platforms available in English and Chinese to tackle the difficulties and challenges of implementing PBL. As the result of this study, we found the PBL platforms were from five countries, serving predominantly K-12 users. According to the services provided, these platforms could be categorized into four types: learning management platforms, content providers, community organizers, communication facilitators, and service provider. Most platforms (10 out of 16) were for learning management, intended to address the major challenge of implementing PBL in regular classrooms. Five out of 16 platforms facilitated the establishment of the professional learning community domestically and internationally for teachers and globally connected classrooms for students. We then conducted the cross-case analysis to see how PBL as a pedagogy was understood through the design of the platforms. In discussion and implication, we discussed the differences between traditional LMS and LMPs, the trend of global classroom connection, and the proposed new skills for teachers and students.

This study has some evident limitations. First, as one of the first studies on PBL e-learning platforms, this study did not involve actual users. Hence the usability and efficiency remain unclear. Despite the researchers' unbiased observation of the platform by accessing the trial versions could provide insights into the platform's capabilities, it is unclear if these platforms offer the facility for teachers to implement PBL-based instruction successfully, as well as for students to adopt and learn efficiently. An empirical study with actual users will be the next step to deepen the understanding of how these platforms can support learning in various classroom contexts, how they coordinate and facilitate the learning process, and to what extent they can help to solve the implementation challenges, especially if the platforms will offer accessibility features in the near future, and how these features will support the learning process of students with learning challenges. Second, how to balance and moderate the support for teachers and foster students' autonomy should be given more thinking and consideration in platform design and improvement. Additionally, studies are also desired to explore how to prepare pre-service and in-service teachers pedagogically and digitally ready to successfully implement PBL with the support of PBL platforms to achieve the optimal learning outcome for students.

Data availability

The data used and analyzed during the current study are available from the corresponding author on reasonable request. Please contact author for data requests.

A good example is the Port of Seattle Units at https://educurious.org/courses/ .

Adnan, M., & Anwar, K. (2020). Online learning amid the COVID-19 pandemic: Students’ perspectives. Online Submission, 2 (1), 45–51.

Google Scholar  

Aksela, M., & Haatainen, O. (2019). Project-based learning (PBL) in practise: active teachers' views of its' advantages and challenges. In Integrated Education for the Real World 5th International STEM in Education Conference Post-Conference Proceedings. Queensland University of Technology.

Al-Balushi, S. M., & Al-Aamri, S. S. (2014). The effect of environmental science projects on students’ environmental knowledge and science attitudes. International Research in Geographical and Environmental Education, 23 (3), 213–227.

Article   Google Scholar  

Aldabbus, S. (2018). Project-based learning: Implementation & challenges. International Journal of Education, Learning and Development, 6 (3), 71–79.

Alverson, J., Schwartz, J., & Shultz, S. (2019). Authentic assessment of student learning in an online class: Implications for embedded practice. College & Research Libraries (C&RL) ,80.

Amini, R., Setiawan, B., Fitria, Y., & Ningsih, Y. (2019). The difference of students learning outcomes using the project-based learning and problem-based learning model in terms of self-efficacy. Journal of Physics: Conference Series, 1387 (1), 012082.

Anazifa, R. D., & Djukri, D. (2017). Project-Based learning and problem-based learning: are they effective to improve student’s thinking skills? Jurnal Pendidikan IPA Indonesia, 6 (2), 346–355.

Barron, B., & Darling-Hammond, L. (2008). Teaching for meaningful learning: A review of research on inquiry-based and cooperative learning book excerpt . George Lucas Educational Foundation.

Blount, E. V. (2016). A qualitative phenomenological study exploring lived experiences of employees and managers and their engagement with e-learning courses . Northcentral University.

Brassler, M., & Dettmers, J. (2017). How to enhance interdisciplinary competence—interdisciplinary problem-based learning versus interdisciplinary project-based learning. Interdisciplinary Journal of Problem-Based Learning, 11 (2), 12.

Cavus, N., Sani, A. S., Haruna, Y., & Lawan, A. A. (2021). Efficacy of social networking sites for sustainable education in the era of COVID-19: A systematic review. Sustainability, 13 (2), 808.

Choudhury, S., & Pattnaik, S. (2020). Emerging themes in e-learning: A review from the stakeholders’ perspective. Computers & Education, 144 , 103657.

Cleland, D. I. (2007). Project management: strategic design and implementation . McGraw-Hill Education.

Connaway, L. S., & Radford, M. L. (2016). Research methods in library and information science . ABC-CLIO.

Culclasure, B. T., Longest, K. C., & Terry, T. M. (2019). Project-based learning (Pjbl) in three southeastern public schools: Academic, behavioral, and social-emotional outcomes. Interdisciplinary Journal of Problem-Based Learning, 13 (2), 5.

Duke, N. K., Halvorsen, A. L., Strachan, S. L., Kim, J., & Konstantopoulos, S. (2021). Putting PjBL to the Test: The Impact of Project-Based Learning on Second Graders’ Social Studies and Literacy Learning and Motivation in Low-SES School Settings. American Educational Research Journal, 58 (1), 160–200.

Dziuban, C., Graham, C. R., Moskal, P. D., Norberg, A., & Sicilia, N. (2018). Blended learning: The new normal and emerging technologies. International Journal of Educational Technology in Higher Education, 15 (1), 1–16.

Farhan, M., Aslam, M., Jabbar, S., & Khalid, S. (2018). Multimedia based qualitative assessment methodology in eLearning: Student teacher engagement analysis. Multimedia Tools and Applications, 77 (4), 4909–4923.

Fauzia, N. L. U., & Kelana, J. B. (2021). Natural science problem solving in elementary school students using the project based learning (PjBL) model. Jurnal Ilmiah Sekolah Dasar, 4 (4), 596–603.

From, J. (2017). Pedagogical digital competence-between values. Knowledge and Skills. Higher Education Studies, 7 (2), 43–50.

Guha, R., Wagner, T., Darling-Hammond, L., Taylor, T., & Curtis, D. (2018). The Promise of Performance Assessments: Innovations in High School Learning and Higher Education Admissions.

Guo, P., Saab, N., Post, L. S., & Admiraal, W. (2020). A review of project-based learning in higher education: Student outcomes and measures. International Journal of Educational Research, 102 , 101586.

Habók, A., & Nagy, J. (2016). In-service teachers’ perceptions of project-based learning. Springerplus, 5 (1), 1–14.

Harris, A., & Johns, A. (2021). Youth, social cohesion and digital life: From risk and resilience to a global digital citizenship approach. Journal of Sociology, 57 (2), 394–411.

Harris, J. (2016). Inservice teachers’ TPACK development: Trends, models, and trajectories. In M. Herring, M. Koehler, & P. Mishra (Eds.), Handbook of technological pedagogical content knowledge for educators (2nd ed., pp. 191–205). Routledge.

Iwamoto, D. H., Hargis, J., & Vuong, K. (2016). The effect of project-based learning on student performance: An action research study. International Journal for the Scholarship of Technology Enhanced Learning, 1 (1), 24–42.

JISC Digital Media. (2014). Guide introduction toe-learning. Retrieved from http://www.jiscdigitalmedia.ac.uk/guide/introduction-to-elearning .

Kassymova, G., Akhmetova, A., Baibekova, M., Kalniyazova, A., Mazhinov, B., & Mussina, S. (2020). E-Learning environments and problem-based learning. International Journal of Advanced Science and Technology, 29 (7), 346–356.

Koehler, M., & Mishra, P. (2009). What is technological pedagogical content knowledge (TPACK)? Contemporary Issues in Technology and Teacher Education, 9 (1), 60–70.

Koehler, M. J., Mishra, P., & Yahya, K. (2007). Tracing the development of teacher knowledge in a design seminar: Integrating content, pedagogy and technology. Computers & Education, 49 (3), 740–762.

Kokotsaki, D., Menzies, V., & Wiggins, A. (2016). Project-based learning: A review of the literature. Improving Schools, 19 (3), 267–277.

Krajcik, J. S., & Blumenfeld, P. C. (2006). Project-based learning (pp. 317-34). na. https://knilt.arcc.albany.edu/images/4/4d/PBL_Article.pdf

Krajcik, J. S., Czerniak, C. M., Czerniak, C. L., & Berger, C. F. (2003). Teaching science in elementary and middle school classrooms: A project-based approach . McGraw-Hill Humanities.

Kraleva, R., Sabani, M., & Kralev, V. (2019). An analysis of some learning management systems. International Journal on Advanced Science, Engineering and Information Technology, 9 (4), 1190–1198.

Lam, S. F., Cheng, R. W. Y., & Choy, H. C. (2010). School support and teacher motivation to implement project-based learning. Learning and Instruction, 20 (6), 487–497.

Lara, J. A., Lizcano, D., Martínez, M. A., Pazos, J., & Riera, T. (2014). A system for knowledge discovery in e-learning environments within the European higher education area-application to student data from Open University of Madrid, UDIMA. Computers & Education, 72 , 23–36.

Lee, D., Huh, Y., & Reigeluth, C. M. (2015). Collaboration, intragroup conflict, and social skills in project-based learning. Instructional Science, 43 (5), 561–590.

Lewis, D. G. R., Gerber, E. M., Carlson, S. E., & Easterday, M. W. (2019). Opportunities for educational innovations in authentic project-based learning: Understanding instructor perceived challenges to design for adoption. Educational Technology Research and Development, 67 (4), 953–982.

Mann, B., Li, W., & Besnoy, K. (2021). Digital Divides: K-12 Student Profiles and Online Learning. Education Policy Analysis Archives , 29 (112), n112.

Meeder, H., & Pawlowski, B. (2020). Preparing our students for the real world: the education shift our children and future demand. National Center for College and Career Transitions, Columbia. https://www.nc3t.com/wp-content/uploads/2020/02/Preparing-Our-Students-for-the-Real-World-021720.pdf .

Mentzer, G. A., Czerniak, C. M., & Brooks, L. (2017). An examination of teacher understanding of project-based science as a result of participating in an extended professional development program: Implications for implementation. School Science and Mathematics, 117 (1–2), 76–86.

Meredith, J. R., Shafer, S. M., & Mantel, S. J., Jr. (2017). Project management: A strategic managerial approach . John Wiley & Sons.

Moore, J. L., Dickson-Deane, C., & Galyen, K. (2011). e-Learning, online learning, and distance learning environments: Are they the same? The Internet and Higher Education, 14 (2), 129–135.

Nasir, M., Fakhrunnisa, R., & Nastiti, L. R. (2019). The implementation of project-based learning and guided inquiry to improve science process skills and student cognitive learning outcomes. International Journal of Environmental and Science Education, 14 (5), 229–238.

Owens, A. D., & Hite, R. L. (2022). Enhancing student communication competencies in STEM using virtual global collaboration project based learning. Research in Science & Technological Education, 40 (1), 76–102.

Ramadhani, R., Rofiqul, U. M. A. M., Abdurrahman, A., & Syazali, M. (2019). The effect of flipped-problem based learning model integrated with LMS-google classroom for senior high school students. Journal for the Education of Gifted Young Scientists, 7 (2), 137–158.

Repko, A. F., Szostak, R., & Buchberger, M. P. (2019). Introduction to interdisciplinary studies . Sage Publications.

Rogers, E. M. (2003). Diffusion of innovations (5th ed.). The Free Press.

Rongbutsri, N. (2017). Students using online collaborative tools in problem-oriented project-based learning . Aalborg University.

Stake, R. E. (2013). Multiple case study analysis . Guilford press.

Thomas, J. W. (2000). A review of research on project-based learning. San Rafael, CA: Autodesk Foundation. https://scirp.org/reference/referencespapers.aspx?referenceid=631638

Trisdiono, H., Siswandari, S., Suryani, N., & Joyoatmojo, S. (2019). Multidisciplinary integrated project-based learning to improve critical thinking skills and collaboration. International Journal of Learning, Teaching and Educational Research, 18 (1), 16–30.

Verstegen, D. M., de Jong, N., van Berlo, J., Camp, A., Könings, K. D., van Merriënboer, J. J., & Donkers, J. (2016). How e-learning can support PBL groups: A literature review. Educational Technologies in Medical and Health Sciences Education, 5 , 9–33.

Vinson, L., & Caukin, N. (2021). eLearning for K-12: challenges and solutions. International Journal of the Whole Child, 2021 (6), 27–35.

Virtue, E. E., & Hinnant-Crawford, B. N. (2019). “We’re doing things that are meaningful”: Student perspectives of project-based learning across the disciplines. Interdisciplinary Journal of Problem-Based Learning, 13 (2), 9.

Wilson, K. (2021). Exploring the challenges and enablers of implementing a STEM project-based learning programme in a diverse junior secondary context. International Journal of Science and Mathematics Education, 19 (5), 881–897.

Wu, T. T., & Wu, Y. T. (2020). Applying project-based learning and SCAMPER teaching strategies in engineering education to explore the influence of creativity on cognition, personal motivation, and personality traits. Thinking Skills and Creativity, 35 , 100631.

Zuo, M., Ma, Y., Hu, Y., & Luo, H. (2021). K-12 students’ online learning experiences during COVID-19: Lessons from China. Frontiers of Education in China, 16 (1), 1–30.

Download references

This work was supported by two sources: International Joint Research Project of Huiyan International College, Faculty of Education, Beijing Normal University [Grant Number ICER201902]; The General Program of National Natural Science Foundation of China. Research on Brain Synchronization Mechanisms and Strategies of Multi-person Interaction in STEM Educational Context, Beijing Normal University [Grant Number 62177011].

Author information

Authors and affiliations.

Department of World Languages, Literatures and Cultures, University of North Texas, Denton, TX, USA

Nanxi Meng & Dorian Roehrs

Faculty of Education, Beijing Normal University, Haidian District, Yanbo Building, No.19, Xinjiekouwai St, Beijing, 100875, People’s Republic of China

Yan Dong & Lin Luan

School of Education, Zhejiang University, Hangzhou, China

Department of Foreign Languages, Beijing University of Posts and Telecommunications, Beijing, China

You can also search for this author in PubMed   Google Scholar

Contributions

This research is conducted by four researchers. All four authors contributed to the study conception and design. NM: conceptualization, data curation, methodology, visualization, and writing. YD: funding acquisition, investigation, supervision, reviewing & editing. DR: methodology review, literature review, writing, reviewing, editing. LL: writing, reviewing, editing. All the authors read and approved the final manuscript.

Corresponding author

Correspondence to Yan Dong .

Ethics declarations

Conflict of interest.

There is no conflict of interest in this article from any of the authors.

Ethical approval

Ethical approval APA ethical standards were followed in the conduct of the study, and we received approval from the Beijing Normal University institutional review board.

Consent to participate

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

See Table 2 .

See Table 3 .

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Meng, N., Dong, Y., Roehrs, D. et al. Tackle implementation challenges in project-based learning: a survey study of PBL e-learning platforms. Education Tech Research Dev 71 , 1179–1207 (2023). https://doi.org/10.1007/s11423-023-10202-7

Download citation

Accepted : 20 January 2023

Published : 28 March 2023

Issue Date : June 2023

DOI : https://doi.org/10.1007/s11423-023-10202-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Project-based learning
• E-learning platforms
• Implementation difficulties
• Project management
• Find a journal
• Publish with us
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• BMC Health Serv Res
• PMC10709868

Challenges and strategies for conducting research in primary health care practice: an integrative review

Daiana bonfim.

1 Hospital Israelita Albert Einstein - Albert Einstein Center for Studies, Research, and Practices in Primary Health Care and Networks, Sao Paulo, Brazil

Lorrayne Belotti

Leticia yamawaka de almeida, ilana eshriqui, sofia rafaela maito velasco, camila nascimento monteiro, adelson guaraci jantsch.

2 Executive Secretariat of Organization Open University of the Unified Health System (UNASUS), Brasilia, Brazil

Associated Data

All data generated or analyzed during this study are included in this published article.

Providing accessible and high-quality patient-centered healthcare remains a challenge in many countries, despite global efforts to strengthen primary health care (PHC). Research and knowledge management are integral to enhancing PHC, facilitating the implementation of successful strategies, and promoting the use of evidence-based practices. Practice-based research in primary care (PC-PBR) has emerged as a valuable approach, with its external validity to diverse PHC settings, making it an effective means of translating research findings into professional practice.

To identify challenges and strategies for conducting practice-based research in primary health care services.

An integrative literature review was conducted by searching the PubMed, Embase, Scopus, Web of Science, and Lilacs databases. The research question, guided by the PICo framework, directed the execution of study selection and data extraction. Data analysis followed the RAdAR method's three phases: pre-analysis, data analysis, and interpretation of results.

Out of 440 initially identified articles, 26 met the inclusion criteria. Most studies were conducted in high-income countries, primarily the United States. The challenges and strategies for PC-PBR were categorized into six themes: research planning, infrastructure, engagement of healthcare professionals, knowledge translation, the relationship between universities and health services, and international collaboration. Notable challenges included research planning complexities, lack of infrastructure, difficulties in engaging healthcare professionals, and barriers to knowledge translation. Strategies underscore the importance of adapting research agendas to local contexts, providing research training, fostering stakeholder engagement, and establishing practice-based research networks.

The challenges encountered in PC-PBR are consistent across various contexts, highlighting the need for systematic, long-term actions involving health managers, decision-makers, academics, diverse healthcare professionals, and patients. This approach is essential to transform primary care, especially in low- and middle-income countries, into an innovative, comprehensive, patient-centered, and accessible healthcare system. By addressing these challenges and implementing the strategies, PC-PBR can play a pivotal role in bridging the gap between research and practice, ultimately improving patient care and population health.

Introduction

Despite global efforts toward strengthening primary health care (PHC) in the last 40 years, providing accessible and good quality patient-centered health care is still a challenge to most countries. Recently, the report Operational Framework for Primary Health Care (2020) released by the World Health Organization reinforced the principles of the Astana Declaration highlighting 14 levers that must be simultaneously pulled to promote PHC across the world [ 1 ].

One of those 14 “operational levers” describes the importance of conducting research that is meaningful for PHC: “ Research and knowledge management, including dissemination of lessons learned, as well as the use of knowledge to accelerate the scale-up of successful strategies to strengthen PHC ” [ 1 ] . Although conducting research that meets these premises is not simple, primary care practice-based research (PC-PBR) has become an important vehicle for the development of science in the real world, because of its external validity to other PHC settings and contexts, making knowledge translation easier to put evidence into professional practice [ 2 ].

PC-PBR occurs in the context of patient health care in the community, according to Dolor et al. (2015), resulting in the research questions being primarily generated by the health services to respond to the needs of their territory [ 3 ]. PHC is responsible for serving as the first point of contact for patients, through which all health issues should be addressed. It serves as an ideal setting for conducting practice-based research, encompassing the implementation of innovations and studies aimed at enhancing the quality of care for various health conditions. These conditions span across diverse areas, including mental health [ 4 ] and chronic kidney disease [ 5 ]. Furthermore, it is also pertinent in the context of public health emergencies, such as the COVID-19 pandemic [ 6 ].

One solution to foster this type of research is creating practice-based research networks (PBRNs). Their aim is to bring healthcare professionals, researchers, health managers, and academic institutions together, facilitating partnerships, and providing structure and technical support to healthcare professionals to carry out research projects that are developed and conducted in PHC settings to tackle important aspects of PHC [ 7 , 8 ]. They also help on the job of acquiring funding, capacity building, organizing the necessary logistics to put a research project in place and all sorts of tasks from study design to publication [ 3 , 9 ]. In this way, PBRNs seek to promote a culture of scientific research in an environment originally dedicated to health care [ 10 ] and to answer relevant questions about the local health needs of PHC services. According to Bodenheimer et al. (2005), PBRNs are increasingly seen as institutions that can simultaneously conduct research efficiently and leverage changes in practice [ 11 ], serving as laboratories for approaching important challenges to PHC.

However, a preview study [ 9 ] developed in Canada described some lessons learned to engage PBRLNs present aspects related to the need for continuity in ethics, regular team meetings, enhancing levels of engagement with stakeholders, the need for structural support and recognizing differences in data sharing across provinces.

Even though the literature on PC-PBR is growing, “How to implement a PBRN and how to scale PC-PBR?” and “How can a healthcare service become a setting for knowledge and innovation production?” are two questions still unanswered. Moreover, scenarios with incipient PHC could benefit from evidence-oriented policies and practice-oriented research. To answer these two questions, available information from places that already run PC-PBR projects needs to be systematized around the challenges, obstacles and solutions found by other researchers. Aiming to help researchers from low- and middle-income countries that are willing to produce research in primary care, we performed an integrative review identifying the challenges and strategies for carrying out PC-PBR.

An integrative literature review was performed based on the methodology proposed by Whittemore & Knafl (2005) [ 12 ] that includes (a) identification of the problem, (b) literature search, (c) evaluation, (d) analysis and (e) presentation of results. Differently from a systematic review, the broader focus of an integrative review enables the inclusion of studies using different methodologies (qualitative, quantitative and mixed) in the analysis and supplies the methodological rigor necessary for a broader understanding of one specific phenomenon [ 13 , 14 ].

Literature search

The research question was developed using the PICo framework (Population, Interest and Context). The elements were organized by P - Primary health care (PHC); I - Challenges and Strategies; Co - Practice-based research (PBR); resulting in the guiding question: “What are the challenges and strategies to carry out PBR in PHC?”. Data were collected in February 2022 by a librarian affiliated with the authors' institution from the databases PubMed, Embase, Scopus, Web of Science, and Lilacs. The database selection was conducted to ensure comprehensive coverage of relevant literature, encompassing multidisciplinary and geographical perspectives related to practice-based research in primary care. The search utilized descriptions and keywords from the Medical Subject Headings (MeSH) and Health Science Descriptors (DeCS), combined with the Boolean operators 'AND' and 'OR' (Table ​ (Table1 1 ).

Search strategies, according to the database and Boolean operators

Study selection

Articles in English, Spanish and Portuguese were included, regardless of their publication year. Review studies, essays, letters to the editor, studies conducted in non-PHC settings (e.g., emergency services), and those focused on specific health problems were excluded.

Two researchers independently screened the articles by title and abstract (SRMV e AGJ), and the disagreements were resolved through discussion and mediation by a third author (LB). Following this stage, the studies were read in their entirety by the same two authors. During this phase, any remaining disagreements regarding the final inclusion were examined and decided by the authors. In the study selection phase, the software Rayyan was employed as a tool for managing and screening research articles.

Data extraction

Information was systematically extracted from the selected articles and organized using a custom-designed spreadsheet, enabling the identification of key aspects essential for addressing the research question. These included author names, publication year, study type, study location, research objectives, methodologies employed, study populations, primary internal and external challenges encountered in operationalizing research within primary healthcare, and strategies offered for its effective implementation.

Data synthesis

The review followed a deductive approach that prioritized the extraction and summarization of studies included as the primary objective of the review and synthesis [ 15 ]. This process entails extracting the results from each included paper and categorizing them according to common themes or meanings. These categories are subsequently further organized, allowing for a summary that yields synthesized findings: practical and actionable guidelines suitable for informing policy and formulating strategies [ 16 ].

To achieve this, the data analysis followed the steps established by the three distinct phases of the RADaR method: pre-analysis, data analysis, and interpretation of the results [ 17 ]. In the pre-analysis stage, each article was read, and its information was extracted and stored in a spreadsheet created to summarize all articles included in the study. In the data analysis stage, the content was categorized according to the similarities of the barriers and challenges identified. Finally, in the interpretation of the results, a reflective and critical analysis of the content was conducted, summarizing the content into themes for analysis [ 17 ].

A total of 440 publications were identified in the databases. After excluding duplicate studies ( n =120) and those that did not answer the guiding question ( n =283), 37 studies were read in their entirety. Out of these, 11 were excluded as they did not meet the eligibility criteria. The final sample consisted of 26 studies (Fig.  1 ), with the majority being published in the past two decades and conducted in high-income countries (HICs), primarily in the United States of America ( n =13). Furthermore, a significant proportion of these studies were case studies focused on the medical profession (Table ​ (Table2 2 ).

Flowchart of study selection

Description of the primary studies included in the integrative literature review according to the lead author, year, country, objective, population, and type of study

NR Not Reported

During the data analysis, six overarching themes and 15 subthemes related to the challenges of carrying out PC-PBR emerged. Among these challenges, difficulties regarding research planning were noteworthy, with issues ranging from excessive bureaucracy to challenges in planning and developing a research project. The Engagement of health professionals in research was recognized as one theme encompassing four different subthemes: lack of training and experience in scientific writing; difficulties with foreign languages; previous negative research experiences; and fears of negative impacts on the healthcare team, patients and productivity. Challenges regarding knowledge translation detail the difficulties in applying the knowledge acquired from one article to a change in daily work. Infrastructure issues are related to the location of the health services and how dispersed they can be in one area, the lack of technological tools and the little access to funding resources to sponsor more robust and long-term projects. Finally, a weak relationship between universities and health services can lead to little – or even no – collaboration between research institutes and PHC practices. The lack of international partnerships is finally presented as one main challenge for low- and middle-income countries (LMICs) since such collaborations could be helpful in building capacity for young research centers to address pressing issues in contexts where PHC is still very incipient (Table ​ (Table3 3 ).

Summary of findings on challenges for conducting PC-PBR

The strategies listed in the articles included in this review were organized according to the challenges described in the previous section. The following were highlighted: suggestions related to creating a research agenda adapted to each reality; training strategies to develop research skills; sharing the results with all stakeholders involved, from participants to health managers and decision-makers; and the importance of creating networks for practice-based research (Table ​ (Table4 4 ).

Strategies for conducting PC-PBR

Challenges and strategies for conducting PC-PBR

Research planning.

In this domain, a series of challenges related to designing a research plan are combined, such as developing and refining a research question, designing a strategy for data collection and data analysis, writing and submitting a proposal to the ethics board committee and the amount of time it takes to obtain the approval to start the project [ 8 , 9 , 11 , 18 , 30 , 32 , 35 ]. The time needed to carry out and conclude a study is often very different from the amount of time needed to make decisions in health care. Conducting a study with the length of time necessary to meet the needs for the transformation of health services is a difficult task, since managers and decision-makers may have more immediate expectations and hope for quick solutions to their problems [ 8 ]. To overcome this limitation, it is important that all stakeholders (managers, patients, health professionals, and researchers) are involved in the study, mainly to facilitate the understanding of the steps that one study needs to go through until its publication [ 9 , 18 , 38 ].

Engagement of health professionals in research

Some decision-makers and health managers fear that a research project can cause trouble in the way that a health facility is used to operate, impairing its productivity or even hindering the patients’ trust in the health service [ 8 , 18 , 21 , 30 , 31 , 35 , 36 ]. In addition, many managers see research projects as less important than practice, without acknowledging the possible benefits of research on patient care [ 28 ]. Researchers must bring these issues into debate with health managers and decision-makers so that barriers such as a lack of time dedicated to research, high caseloads limiting the time dedicated to research, and the need for institutional approval to allow professionals to participate in research projects can be overcome [ 26 ]. If this is not done, it will be difficult to create a routine of knowledge production and innovative research that integrates healthcare professionals, patients and researchers to create robust scientific evidence with an impact on the workplace, patient care and the quality of the services provided.

Knowledge translation

This theme, which is known as integrated knowledge translation in the current literature [ 39 ], involves the processes of generating, sharing, and applying knowledge, not necessarily in that specific order [ 8 , 32 ]. In theory, carrying out PC-PBR is a powerful resource to make knowledge translation happen, since research questions are created to answer local needs, relying on the participation of professionals – and sometimes the patients – in practice [ 32 ].

However, one of the barriers to knowledge translation lies in the difficulty of adapting the knowledge to contexts that are distinct from those where one study was held, e.g., results from HIC being translated to LMICs. This reinforces the need to involve all stakeholders in the stages of designing the project to describe the aspects of the context where the research will be held, outlining this information in the discussion section of the article as well, making it easier for the reader to understand its external validity [ 2 , 8 , 30 , 38 ].

The long time span for the publication of the study results in scientific journals, in addition to the high rejection rate, are factors that further delay the process of knowledge translation. Considering the dynamic nature of primary care services, studies should have a broad plan to disseminate results, to implement the evidence in a timely manner [ 30 ].

Infrastructure

Challenges related to infrastructure are frequently found in PC-PBR studies, from the distance between primary care services in rural settings and the difficulty of reaching some services to the often lack of technology resources, such as internet access, and patients’ electronic records [ 8 , 9 , 20 , 23 , 32 , 35 ].

The lack of reliable, sustainable, and systematic funding for PC-PBR research activities is the main obstacle to overcoming these infrastructure limitations and promoting the creation of PC-PBR [ 8 , 10 , 19 , 23 , 27 , 31 , 35 ]. Like every research initiative, PC-PBR needs to be supported with adequate and constant funding. For that reason, researchers must remain attentive and updated to identify funding opportunities [ 18 ].

Healthcare services produce a large volume of data every day. Information about healthcare procedures, prescriptions, patient profile, and all sorts of interactions between the patient and their healthcare providers. However, the quality of the information input and the way it is stored can limit its use [ 9 ]. It is essential for managers and stakeholders to verify how these data have been used, not only how practitioners use them for patient management but also for research, surveillance, and accountability [ 19 , 23 ].

Confidential information should be strictly and safely handled so that no patient information becomes public, allowing its use for research with no harm to the patient or for the practice [ 34 ]. For this purpose, all parties using these data must agree to a common commitment across the PC-PBR network to develop and implement research programs. Ideally, the research priorities should be established by the researchers and managers, with a clear evaluation of the capabilities of each practice, the information systems available and the whole network. When used appropriately, these real-world data can generate new knowledge from practice to improve patient care [ 18 ].

Relationship between universities and health services

Some studies highlighted the strains of integrating universities and health services [ 8 , 18 , 21 ]. The distance between these two scenarios can be explained by several factors: (a) the fact that academic priorities may not reflect the needs of the communities [ 8 ]; (b) weak connections between academia and primary care services [ 19 ]; (c) the lack of a mutual agenda between them combining common interests [ 25 ]; (d) the distance between researchers and health professionals [ 8 ]; and (e) the restricted access to specific research training courses run by universities, apart from formal master’s and doctorate courses [ 21 ]. Such training courses are usually offered during workdays, which limits the participation of those who work full-time as health care providers. Offering postgraduate courses in research aimed at health professionals that take advantage of the students’ experience to generate relevant research questions and new knowledge for healthcare could be transformative both for universities and health services. However, gathering individuals who traditionally work in different sectors is not easy. In addition, creating organizational structures that support primary care-based studies can demand financial resources, time, and people, which are not easily available [ 29 ].

Among the strategies found in the articles to overcome this challenge, it is important that the research questions arise from practice and that the roles of researchers, academics and health professionals are well-defined within the group. In addition, it is important to select a coordinator responsible for managing the research project and the tasks that need to be executed [ 30 , 34 ].

Implementing PC-PBR can bring results both for practice and academia, bringing together different professionals to achieve a common goal of improving patient care. Strengthening the interaction between academia and primary care services can help to promote the sustainable development of research projects in which health professionals can develop innovations in health care that can be studied and tested, creating a virtuous cycle beginning with raising questions from practice, conducting experiments, finding results and producing evidence that can serve the purpose of improving patient care and the health of the population [ 19 ].

Partnerships between countries

Despite this being a topic addressed in only two of the articles under analysis, promoting international partnerships can be a solution to many of the challenges mentioned here. However, such collaborations are not yet a reality for many countries. There is a shortage of international initiatives to promote research courses and training to bring together mentors from HIC and young researchers from LMICs and provide direction for conducting studies in contexts with few resources [ 8 ].

In addition, many professionals from LMICs who are involved in studies or education abroad end up migrating to other countries, contributing to the so-called “brain drain” of skilled professionals and worsening the inequality in scientific production between HICs and LMICs.

Addressing research projects within the local context and exploring opportunities for international collaboration is important enough to foster PBR and guide health professionals in places where universities and research institutes are not yet established. Moreover, it is important to consider the epidemiological profile, cultural aspects, and social determinants of health in every scenario involved when an international collaboration is planned. The different contexts of practice can enrich the research and establish comparisons that can be decisive for international scientific advancement [ 8 ].

The challenges and strategies for the implementation of PC-PBR indicate operational, structural, and political issues. One of the key aspects learned about planning a PC-PBR study is to identify and include all stakeholders (patients, employees, doctors and administration) in the development phase of the project, allowing for discussions about the study design and its implementation phases. This approach must become an integral part of the study, being comprehensive to addressing barriers to participation, obtain data, analyze and interpret the results and, finally, discuss its findings and implications. Additionally, planning data collection that demands little effort from health professionals can strengthen the study’s realization and the involvement of everyone.

In this context, it is important to emphasize that all challenges are even more pronounced in LMICs. In this regard, efforts are being made towards decolonization [ 40 ], encouraging research that validates the context and perspectives of local thinkers, thereby expanding the discussion to generate and incorporate evidence into real scenarios that value the knowledge of communities, healthcare professionals, policymakers, and researchers in LMICs. Therefore, the present study aimed to synthesize the challenges and strategies that underlie this discussion, but a gap was identified in terms of the production of this discussion in LMICs.

To address the issue of limited international collaborations in LMICs, it is crucial to explore targeted implications and strategies to surmount this constraint. Some viable strategies involve providing training and education in cultural sensitivity, thereby enhancing the efficacy of these partnerships. While international collaboration typically prioritizes partnerships with high-income countries, LMICs can also explore collaborations with other LMICs. Sharing knowledge, best practices and resources with neighboring countries facing similar challenges can result in mutually advantageous outcomes.

PC-PBR only happens if the professionals who are directly involved in patient care and health service management are integrated as part of the team of researchers, not just as the subjects of the research [ 8 , 36 ]. Although it is a great challenge, training healthcare professionals to conduct research in primary care is fundamental for the success of these projects [ 23 , 24 ].

Alternative research approaches, such as implementation research, have advanced and grown as new strategies to reduce the gap between research and practice, mainly because they systematically approach the factors that contribute to this gap, understanding the context and identifying barriers and solutions for delivering sustainable and effective health care [ 41 ]. Thus, to make progress in overcoming these structural barriers it is important to understand the essential pieces of the research process, without which a project will likely die prematurely. One of these elements is the minimal infrastructure needed for PC-PBR research projects to be long-lasting and sustainable [ 9 , 23 ].

The studies under analysis point out that the most promising way for this to happen is through collaboration between primary care services, universities, and research institutes. In addition, these collaborations can provide training in research skills for health professionals, creating an environment conducive to exchanging experiences, ideas, and questions about the practice. All these suggestions will help to create a research agenda oriented toward solving real issues related to taking care of patients in primary care, which is the main objective of conducting PC-PBR [ 8 ].

The distance between universities and primary care settings is recurrently cited. This issue reinforces the idea that there is a place where knowledge is produced (universities and academia) that is different from the places where health care occurs. In other words, primary care is seen as a place where scientific evidence produced by academia is put into practice.

Conducting scientific research within primary care practices is innovative and can create ruptures and conflicts when it affects the way the job is done or when it takes people out of their comfort zones. By placing health professionals—and at times, patients—as agents of research production, PC-PBR can change the way new knowledge is produced. If knowledge is traditionally produced in academia and then taken as a truth by the place where patient care occurs, PC-PBR can not only generate new knowledge to change professional practice but also bring new evidence to change the way academia works, guiding new research that is better aligned with reality [ 34 ].

In some countries, a more horizontal construction of new evidence and knowledge translation can be seen between academia and healthcare practice. In Australia, for example, PBR protocols are designed to build a sustainable collaboration between a PBRN and an Advanced Center of Research and Translation in Health to build a research platform for planning, conducting and translating research evidence to improve care across the healthcare spectrum [ 42 ].

Aligned with the need for partnership between universities and practices, international collaborations are also an opportunity to guide professionals in places where universities and research institutes are not yet established. Cases such as Australia and New Zealand, where two PBR networks were established to encourage research in the area of osteopathy, show that PBRN has the potential to facilitate the access of professional researchers and clinics that are interested in collaborating with clinical tests and, thus, offer the scientific community an opportunity to conduct research with different methodologies in diverse contexts [ 42 ].

Regarding the difficulties in engaging health professionals in PC-PBR, some examples listed in the articles were little experience in scientific writing, difficulties reading articles in foreign languages, limited self-trust and lack of training to start and conduct studies. Thus, studies recommend that universities and research institutes organize training courses to develop research skills and exchange experiences to determine shared research priorities [ 8 ].

Although essential, the development of research skills is not enough for professionals to engage with and incorporate studies into their places of practice. For PC-PBR projects to advance, leadership is necessary to influence policymakers and managers and advocate for studies to be directly connected with the practice where health care happens.

The majority of the selected studies highlighted the medical category in the discussion about PBR. However, it is important to expand the professional composition of PC-PBR beyond and consider other categories to organize more participative and multidisciplinary studies. All health professionals must be invited to interact and collaborate with scientific activities and implement new projects. The inclusion of all health professionals, including community health workers, nursing assistants, and dental hygienists, who are commonly found in LMICs, can improve the development of research projects that will better take into consideration the patients’ and the territory’s needs [ 8 ].

Implementing PC-PBR goes beyond research production, since the results of the studies produced by researchers, health professionals, users and managers, in addition to the lessons learned, are shared with the health service where the study was held, bringing greater transparency to the entire process and motivating more health professionals to actively participate in future research projects [ 38 ].

Limitations

This review was limited to the literature that reported lessons learned and experiences conducting PC-PBR since few empirical studies with primary data from practice were found. Additionally, there is little representation from LMICs. This limits the conclusions of this review to the contexts described herein, i.e., HIC, where PHC already has a solid structure and a robust research production. Exploring studies performed in PC-PBR networks and identifying their strengths and weaknesses would be a step forward in this sense, but it would demand greater operational efforts. However, this is the first review that is necessary for the advancement of primary care research mainly in LMIC.

The challenges for implementing PBR are similar in the contexts analyzed, showing that turning one place that was originally designed for delivering primary care into a place of knowledge production is not a trivial task. The benefits depicted in the studies show that transforming the traditional methods of knowledge production and translation through PC-PBR can generate a virtuous cycle, providing criticism and reflection about the practice and generating innovations and new knowledge to improve healthcare and patients’ health and well-being.

Additionally, the found strategies point to the need for lasting and systemic actions involving health managers, decision-makers, academics, different types of health professionals and patients, aiming to transform PHC practice in the long term. Despite being more the exception than the rule, PC-PBR has the potential to transform a PHC system that is still under development into an innovative, socially accountable, more comprehensive, accessible, and patient-centered healthcare approach. Furthermore, recognizing the transformative potential of PC-PBR, it becomes imperative to explore strategies for scaling these practices and approaches, ultimately having a broader and more profound impact on the entire primary healthcare system.

Acknowledgment

Not applicable.

Authors’ contributions

Conception and planning of the study: DB and AGJ. Writing the main manuscript text: DB, LB, LYA, IEO, SRMV, CNM, AGJ. Analysis and interpretation: DB, LB, LYA, IEO, SRMV, CNM, AGJ. All the authors read and gave final approval for the final version to be published and agreed to be accountable for all aspects of the work.

The study received no funding.

Availability of data and materials

Declarations.

The authors declare no competing interests.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

• Work & Careers
• Life & Arts
• Currently reading: Business school teaching case study: Unilever chief signals rethink on ESG
• Business school teaching case study: can green hydrogen’s potential be realised?
• Business school teaching case study: how electric vehicles pose tricky trade dilemmas
• Business school teaching case study: is private equity responsible for child labour violations?

Business school teaching case study: Unilever chief signals rethink on ESG

• Business school teaching case study: Unilever chief signals rethink on ESG on x (opens in a new window)
• Business school teaching case study: Unilever chief signals rethink on ESG on facebook (opens in a new window)
• Business school teaching case study: Unilever chief signals rethink on ESG on linkedin (opens in a new window)
• Business school teaching case study: Unilever chief signals rethink on ESG on whatsapp (opens in a new window)

Gabriela Salinas and Jeeva Somasundaram

Roula Khalaf, Editor of the FT, selects her favourite stories in this weekly newsletter.

In April this year, Hein Schumacher, chief executive of Unilever, announced that the company was entering a “new era for sustainability leadership”, and signalled a shift from the central priority promoted under his predecessor , Alan Jope.

While Jope saw lack of social purpose or environmental sustainability as the way to prune brands from the portfolio, Schumacher has adopted a more balanced approach between purpose and profit. He stresses that Unilever should deliver on both sustainability commitments and financial goals. This approach, which we dub “realistic sustainability”, aims to balance long- and short-term environmental goals, ambition, and delivery.

As a result, Unilever’s refreshed sustainability agenda focuses harder on fewer commitments that the company says remain “very stretching”. In practice, this entails extending deadlines for taking action as well as reducing the scale of its targets for environmental, social and governance measures.

Such backpedalling is becoming widespread — with many companies retracting their commitments to climate targets , for example. According to FactSet, a US financial data and software provider, the number of US companies in the S&P 500 index mentioning “ESG” on their earnings calls has declined sharply : from a peak of 155 in the fourth quarter 2021 to just 29 two years later. This trend towards playing down a company’s ESG efforts, from fear of greater scrutiny or of accusations of empty claims, even has a name: “greenhushing”.

Test yourself

This is the fourth in a series of monthly business school-style teaching case studies devoted to the responsible business dilemmas faced by organisations. Read the piece and FT articles suggested at the end before considering the questions raised.

About the authors: Gabriela Salinas is an adjunct professor of marketing at IE University; Jeeva Somasundaram is an assistant professor of decision sciences in operations and technology at IE University.

The series forms part of a wider collection of FT ‘instant teaching case studies ’, featured across our Business Education publications, that explore management challenges.

The change in approach is not limited to regulatory compliance and corporate reporting; it also affects consumer communications. While Jope believed that brands sold more when “guided by a purpose”, Schumacher argues that “we don’t want to force fit [purpose] on brands unnecessarily”.

His more nuanced view aligns with evidence that consumers’ responses to the sustainability and purpose communication attached to brand names depend on two key variables: the type of industry in which the brand operates; and the specific aspect of sustainability being communicated.

In terms of the sustainability message, research in the Journal of Business Ethics found consumers can be less interested when product functionality is key. Furthermore, a UK survey in 2022 found that about 15 per cent of consumers believed brands should support social causes, but nearly 60 per cent said they would rather see brand owners pay taxes and treat people fairly.

Among investors, too, “anti-purpose” and “anti-ESG” sentiment is growing. One (unnamed) leading bond fund manager even suggested to the FT that “ESG will be dead in five years”.

Media reports on the adverse impact of ESG controversies on investment are certainly now more frequent. For example, while Jope was still at the helm, the FT reported criticism of Unilever by influential fund manager Terry Smith for displaying sustainability credentials at the expense of managing the business.

Yet some executives feel under pressure to take a stand on environmental and social issues — in many cases believing they are morally obliged to do so or through a desire to improve their own reputations. This pressure may lead to a conflict with shareholders if sustainability becomes a promotional tool for managers, or for their personal social responsibility agenda, rather than creating business value .

Such opportunistic behaviours may lead to a perception that corporate sustainability policies are pursued only because of public image concerns.

Alison Taylor, at NYU Stern School of Business, recently described Unilever’s old materiality map — a visual representation of how companies assess which social and environmental factors matter most to them — to Sustainability magazine. She depicted it as an example of “baggy, vague, overambitious goals and self-aggrandising commitments that make little sense and falsely suggest a mayonnaise and soap company can solve intractable societal problems”.

In contrast, the “realism” approach of Schumacher is being promulgated as both more honest and more feasible. Former investment banker Alex Edmans, at London Business School, has coined the term “rational sustainability” to describe an approach that integrates financial principles into decision-making, and avoids using sustainability primarily for enhancing social image and reputation.

Such “rational sustainability” encompasses any business activity that creates long-term value — including product innovation, productivity enhancements, or corporate culture initiatives, regardless of whether they fall under the traditional ESG framework.

Similarly, Schumacher’s approach aims for fewer targets with greater impact, all while keeping financial objectives in sight.

Complex objectives, such as having a positive impact on the world, may be best achieved indirectly, as expounded by economist John Kay in his book, Obliquity . Schumacher’s “realistic sustainability” approach means focusing on long-term value creation, placing customers and investors to the fore. Saving the planet begins with meaningfully helping a company’s consumers and investors. Without their support, broader sustainability efforts risk failure.

Questions for discussion

Read: Unilever has ‘lost the plot’ by fixating on sustainability, says Terry Smith

Companies take step back from making climate target promises

The real impact of the ESG backlash

Unilever’s new chief says corporate purpose can be ‘unwelcome distraction ’

Unilever says new laxer environmental targets aim for ‘realism’

How should business executives incorporate ESG criteria in their commercial, investor, internal, and external communications? How can they strike a balance between purpose and profits?

How does purpose affect business and brand value? Under what circumstances or conditions can the impact of purpose be positive, neutral, or negative?

Are brands vehicles by which to drive social or environmental change? Is this the primary role of brands in the 21st century or do profits and clients’ needs come first?

Which categories or sectors might benefit most from strongly articulating and communicating a corporate purpose? Are there instances in which it might backfire?

In your opinion, is it necessary for brands to take a stance on social issues? Why or why not, and when?

Climate Capital

Where climate change meets business, markets and politics. Explore the FT’s coverage here .

Are you curious about the FT’s environmental sustainability commitments? Find out more about our science-based targets here

Promoted Content

Explore the series.

Follow the topics in this article

• Sustainability Add to myFT
• Impact investing Add to myFT
• Corporate governance Add to myFT
• Corporate social responsibility Add to myFT
• Business school case Add to myFT

International Edition

Identify Motivation and Challenges in Data Modeling

After completing this lesson, you will be able to:

• Identify motivation and challenges in data modeling

Motivation for Data Modeling

Difference in values.

Values collected from information processing systems often drive business decisions. Therefore, numerous reports are distributed. But where do the values come from and how reliable are they?

Imagine the decision-maker asking for the total quantity that has been delivered to a specific customer, getting two different answers:

• The storage department states that they sold 14 KG.
• The sales department states that the company sold 15 KG.

Without a central data collection strategy, each department relies on their own calculations. If every department keeps their own, independent database, based on the same original business process, this is called a silo.

Silos are multiple copies of the same data source with different, independent steps of consolidation, harmonization, and enhancement.

The problems with silos are:

• Misunderstandings in discussions.
• Aggregation of errors.
• Different beliefs about the truth and the recommended behavior.
• Differences in values between different public reports.

Why do different departments get different results? Let's look at an example with possible reasons for the differences:

• Different database : Different source tables (such as an aggregated source table versus a details source table), different filter values, or different times, or different source systems regularly lead to different extraction results.
• Different actuality of values : If, at an early stage, the wrong data was extracted from the source system that was corrected later. It's important to update the values in the reporting system.
• Different times of extraction : Values in the source vary. New orders might be added, or status information may change. Then, values may differ depending on whether they're extracted in the morning or in the evening.
• Different calculation definitions : How is a margin defined? Which costs must be distracted? Is a percentage value based on kilograms or liters?
• Different steps of calculation : Is an aggregation over different products calculated before a unit conversion is calculated on an average conversion factor? Or is the conversion carried out with a product-specific conversion factor before values are aggregated? Is a share (percentage value) calculated before or after a unit conversion is carried out?
• Different times of calculation : If conversion factors change, different conversion factors lead to different results.
• No good quality management : You've no control over the reliability of the output.

Data Modeling Directive

Aspects of a data modeling directive.

Many companies decide to implement a data warehouse, such as SAP BW/4HANA, with a central storage of all enterprise information, and fast information processing.

However, even the best software solution must be accompanied by a good data modeling directive.

A data modeling directive is a guideline regarding what must be done: where, when, with what data, and in which way?

A data modeling directive has to address the following:

• What business processes are in focus?
• Are values needed for the entire enterprise, or for a local subsidiary?
• What fields of data management are intended? Reporting actual values, predictive analysis, and operation planning?
• To what level of detail are the values needed?
• Are current values (up to the last second) essential? Or is a latency (a delay), acceptable? Is it sufficient to process data once per day?
• Are historical values required? How far back? If values change, is it necessary to identify obsolete values?
• In calculations, are intermediate results needed as well?

A data modeling directive must address what to do with the data:

• Are the values read, copied, processed, or presented?
• Are selection processes used to reduce the set of data?
• Are tables joined with other tables?
• Are values transformed? Are they harmonized?
• Which values must be checked, for instance for referential integrity, against duplicates, for the correct data format, against mathematical errors such as a sum of more than 100%?
• Is a result presented as a result on the screen, or is it permanently stored?
• Is data archived or deleted?

A data modeling directive has to address when, and how often, these processes occur. Is it:

• On demand? (Whenever a business user is interested in the values).
• At a predefined point in time?
• Periodically, for instance daily or monthly.
• Every time the data has changed in the source?

A data modeling directive has to address where data is read and processed:

• Which source systems are used as a database, and which target systems are used to store values?
• Which tables or views contain the database?
• Are calculations applied in the database, or in the application level?
• Which database is used?
• Which objects are needed to meet the approved requirements?

Finally, a data modeling directive must address how the data is treated:

• Are values from imports using flat files, a database connection, or other technologies?
• What are the join conditions?
• What are the filter conditions?
• Which calculations must be carried out?
• How are plan values derived?
• Which currencies or units are used, and how can values be converted from one currency to another one?
• How are values aggregated?

For instance, you must define how key figures from the source are made available. For example, a directive could suggest a raw hub with basic key figures as they're imported from the data source. Business users can then create their own calculations and restrictions on those key figures.

A core hub is another solution that provides predefined, calculated, or restricted key figures for typical use cases. In this case, a data model must define how the new key figures are derived.

You can also provide both options.

Modeling Strategy with Different Reporting Requirements

Different reporting requirements.

It's important to realize that different users in a company may have different requirements related to business reports.

Watch the video to learn about the different reporting requirements for ITelO.

Strategy - Combining Different Requirements

Sometimes, there are good ways to let two requirements coexist:

• Choose characteristics for drill-down.
• Add calculations.
• Create filters for the Top N results.
• Create highlighting for values below an individually defined target value.
• In SAP Analytics Cloud, business users can influence the graphical display.
• Provide a strategic report with a long history and another report with more details for the current year or month.
• Allow the users to choose one of two different hierarchy versions in one report.
• List two alternative key figures, such as quantity in KG and quantity in number of pieces.
• Provide a German HR report without information about the number of sick days per employee, and an American report that orders employees by number of days on leave.
• In a customer relationship management project, provide a local report showing the number of recurring orders. In a sales project, create another local report showing the number of open orders. However, a global strategy should detect that both use the number of orders, and provide a common ground with harmonized order key figures for both.

Strategy - Target Prioritization

Sometimes, you may decide to exclude certain content or keep it out of the project focus. There might be different reasons, such as:

• It's impossible to satisfy both targets. For example, it’s impossible to have complex algorithms that are easy to understand at first sight.
• It's time-consuming and expensive to produce such results.
• There are legal or compliance barriers.
• There are global standards. For example, if the global decision was made in favor of showing a standard product categorization based on the first digits of their technical names, other categorizations aren't supported.
• For users, it's confusing to have different versions.
• The data sources can't be trusted.
• Other targets have a higher priority.

If there are conflicts, your priorities must be as follows:

• Correct values (true, in line with legal requirements, understandable)
• Required functionality
• Flexibility (being open for future requirements)
• Reporting performance
• Staging performance (correlates with low data volume)

If the conditions change, the targets might be reevaluated, and later projects may be set up to accommodate the targets that were left out.

To avoid confusion, you must aim for:

• Valid values.
• Comprehensible data flows.
• A transparent data origin.
• A well-structured model.
• Clear responsibilities.
• Standard calculations and enterprise-wide available data.
• Single point of truth.
• Access control (authorization).

Therefore, Cathy's requirement about integrating dynamic values from the Internet is out of scope.

Case Study - Start

Analyze business segments.

Let's first focus on analyzing the business segments of a company, because business segments have different processes, different issues, and different reporting requirements. You may hear the following questions:

What are your business processes?

What are the main reporting requirements of each business?

• Where are the issues relating to analysis and reporting?

Where are strategic reports relevant, where is the focus on real-time insight, and where is harmonization across systems an issue?

In the example in the figure above, you identify storage and sales as business segments with urgent requests for new reports. To find reporting requirements, start by analyzing the business process of the business segment and the issues.

In storage, the business process involves an automatically controlled inflow and outflow of goods. You may not realize ahead of time that there's a danger of insufficient material stock. You need real time data analysis with good performance on a detailed level.

The sales team wants to integrate data from different source systems and therefore wants to generate reliable harmonized data. In the sales business process, orders are placed in different currencies. The different currencies must be converted to the company's currency. To guarantee reliable data, the results must be saved permanently.

Let's have a more detailed look at the requirements for sales in ITelO.

Let's have a more detailed look at the requirements for storage in ITelO.

Separate Business Segments

As a first step, the business segments were analyzed and in this case storage and sales are the two most important business segments to focus on for ITelO.

As a second step, you have to define how the organization is divided. In this case study, the main distinction is the origin of the data. This distinction is important because ITelO and Retailer King 3000 have different data models. Moreover, each organization consists of different organizational units. Each organization unit operates different storage locations (which is especially important for storage, but also for sales).

The third step is to investigate how this separation is represented in the technical systems.

Each business segment stores its data in a set of database tables. Make a note of any tables that are used by more than one business segment. In our case study, the product master data tables are used by both storage and sales.

Identify which field represents the organization unit. If you want to extract values only for one organization unit, apply a filter on this field.

When inspecting the relevant tables, ask yourself the following questions:

• What fields always have unique values? What fields contain values that can be repeated in different rows?
• Is there a check table for a field? Can you only enter values that are listed in other tables?
• What values are likely to change and when?

Bringing Together Separate Segments

In a fourth step, you define a target landscape. In the ITelO case study, the main task is to combine storage and sales values from different sources. For example, ITelO wants to see the sum of all sales volumes for the same category.

To define this task on a business level, check what separation must remain and what level of integration is intended.

To understand the integration task, first check what differences exist between corresponding business processes. Then, check the differences in the data model (table names, table fields, allowed values, and behavior when data changes).

After investigating the differences, you can design a model to integrate the data by following these steps (this process will be defined in more detail later in this course):

Homogenize the master data.

In this case study, it’s easy to integrate product master data because Retailer King 3000 sells the same products and uses the same product ID. However, Retailer King 3000 uses different master data such as categories and product prices. Decide which system is the leading system, that is, from which system the price and category information is taken. Alternatively, both options can be presented as a choice to the business user.

Prepare transactional data.

In this case study, storage and sales remain separate areas of reporting. For storage, both sources have tables of similar design, and a view that brings together both sources is sufficient. For the sales data, you must load harmonized values. Map data from different systems to the same format, then add a source distinguishing field to the key of each table.

Define a query, view, or report.

Define a query, a view, or a report that aggregates the sales values across both sources and joins the master data, and other external data.

Extra tasks involve the integration of other sources. In our case study, you want to compare sales and purchase amounts.

Log in to track your progress & complete quizzes

This paper is in the following e-collection/theme issue:

Published on 23.5.2024 in Vol 8 (2024)

Barriers to Implementing Registered Nurse–Driven Clinical Decision Support for Antibiotic Stewardship: Retrospective Case Study

Authors of this article:

Original Paper

• Elizabeth R Stevens 1 , MPH, PhD   ; 
• Lynn Xu 1 , MPH   ; 
• JaeEun Kwon 1 , MPP   ; 
• Sumaiya Tasneem 1 , MPH   ; 
• Natalie Henning 1 , MPH   ; 
• Dawn Feldthouse 1 , RN-BC, MSN   ; 
• Eun Ji Kim 2 , MSc, MS, MD   ; 
• Rachel Hess 3, 4 , MS, MD   ; 
• Katherine L Dauber-Decker 2 , PhD   ; 
• Paul D Smith 5 , MD   ; 
• Wendy Halm 6, 7 , DNP   ; 
• Pranisha Gautam-Goyal 2 , MD   ; 
• David A Feldstein 6 , MD   ; 
• Devin M Mann 1, 8 , MS, MD  

1 Department of Population Health, New York University Grossman School of Medicine, New York, NY, United States

2 Northwell, New Hyde Park, NY, United States

3 Department of Population Health Sciences, University of Utah, Salt Lake City, UT, United States

4 Department of Internal Medicine, University of Utah, Salt Lake City, UT, United States

5 Department of Family Medicine and Community Health, University of Wisconsin School of Medicine and Public Health, Madison, WI, United States

6 Department of Medicine, University of Wisconsin School of Medicine and Public Health, Madison, WI, United States

7 University of Wisconsin-Madison School of Nursing, Madison, WI, United States

8 Department of Medicine, New York University Langone, New York, NY, United States

Corresponding Author:

Elizabeth R Stevens, MPH, PhD

Department of Population Health

New York University Grossman School of Medicine

180 Madison Ave

New York, NY, 10016

United States

Phone: 1 6465012558

Email: [email protected]

Background: Up to 50% of antibiotic prescriptions for upper respiratory infections (URIs) are inappropriate. Clinical decision support (CDS) systems to mitigate unnecessary antibiotic prescriptions have been implemented into electronic health records, but their use by providers has been limited.

Objective: As a delegation protocol, we adapted a validated electronic health record–integrated clinical prediction rule (iCPR) CDS-based intervention for registered nurses (RNs), consisting of triage to identify patients with low-acuity URI followed by CDS-guided RN visits. It was implemented in February 2022 as a randomized controlled stepped-wedge trial in 43 primary and urgent care practices within 4 academic health systems in New York, Wisconsin, and Utah. While issues were pragmatically addressed as they arose, a systematic assessment of the barriers to implementation is needed to better understand and address these barriers.

Methods: We performed a retrospective case study, collecting quantitative and qualitative data regarding clinical workflows and triage-template use from expert interviews, study surveys, routine check-ins with practice personnel, and chart reviews over the first year of implementation of the iCPR intervention. Guided by the updated CFIR (Consolidated Framework for Implementation Research), we characterized the initial barriers to implementing a URI iCPR intervention for RNs in ambulatory care. CFIR constructs were coded as missing, neutral, weak, or strong implementation factors.

Results: Barriers were identified within all implementation domains. The strongest barriers were found in the outer setting, with those factors trickling down to impact the inner setting. Local conditions driven by COVID-19 served as one of the strongest barriers, impacting attitudes among practice staff and ultimately contributing to a work infrastructure characterized by staff changes, RN shortages and turnover, and competing responsibilities. Policies and laws regarding scope of practice of RNs varied by state and institutional application of those laws, with some allowing more clinical autonomy for RNs. This necessitated different study procedures at each study site to meet practice requirements, increasing innovation complexity. Similarly, institutional policies led to varying levels of compatibility with existing triage, rooming, and documentation workflows. These workflow conflicts were compounded by limited available resources, as well as an implementation climate of optional participation, few participation incentives, and thus low relative priority compared to other clinical duties.

Conclusions: Both between and within health care systems, significant variability existed in workflows for patient intake and triage. Even in a relatively straightforward clinical workflow, workflow and cultural differences appreciably impacted intervention adoption. Takeaways from this study can be applied to other RN delegation protocol implementations of new and innovative CDS tools within existing workflows to support integration and improve uptake. When implementing a system-wide clinical care intervention, considerations must be made for variability in culture and workflows at the state, health system, practice, and individual levels.

Trial Registration: ClinicalTrials.gov NCT04255303; https://clinicaltrials.gov/ct2/show/NCT04255303

Introduction

Antibiotic resistance is a major public health risk, with more than 35,000 deaths each year in the United States due to antibiotic-resistant bacterial infections [ 1 , 2 ]. Overprescribing and misuse of antibiotics for upper respiratory infections (URIs) remain the most significant combined factors causing antibiotic resistance [ 3 , 4 ]. In the United States, up to 50% of all outpatient antibiotic prescriptions for URIs are inappropriate [ 5 , 6 ].

An estimated 80%-90% of antibiotic prescribing occurs in outpatient settings, such as doctors’ offices, urgent care facilities, and emergency departments [ 7 - 9 ]. From 1996 to 2010, 72% of adult patients in primary care with a diagnosis of acute bronchitis received antibiotics contrary to guideline recommendations against antibiotic treatment, and prescription rates actually increased during this time frame [ 10 ]. Patients with sore throats received antibiotics 61% of the time when the prevalence of group A streptococcus, the only clear indication for antibiotics, was only 10% in adults [ 11 ].

By providing real-time evidence-based data to assist providers (physicians, nurse practitioners, and physician assistants) in estimating the likelihood of a patient having either pneumococcal pneumonia or group A streptococcus, electronic health record (EHR)–integrated clinical prediction rules (iCPRs) can help address prescriber-level barriers to antibiotic stewardship and reduce antibiotic prescribing for URIs in primary care [ 12 - 15 ]. Indeed, CPRs have already been validated to successfully distinguish between viral and bacterial respiratory infections [ 16 - 18 ].

While potentially effective, there is low uptake of the iCPR tools among physicians in primary care practices, thus indicating implementation barriers to antibiotic stewardship iCPRs among physicians [ 19 ]. This outcome is consistent with other literature, indicating that physicians perceive antibiotic stewardship as onerous and would require substantial assistance to change their antibiotic prescribing behaviors [ 20 ]. Due to these limitations associated with the physician-driven iCPR implementation model, such as “alert fatigue” and time constraints [ 21 , 22 ], the iCPR intervention was adapted so antibiotic stewardship tasks could be delegated to other qualified members of the medical team.

A registered nurse (RN)–driven implementation model of iCPR for low-acuity URIs has the potential to be an effective alternative to the physician-driven implementation model. RNs have demonstrated equivalent symptom resolution compared to physicians when using protocols to improve ambulatory care across a number of chronic diseases [ 23 ] as well as the treatment of acute minor illnesses [ 24 , 25 ]. Therefore, the iCPR intervention was adapted for RNs to include the identification of patients with low-acuity URI followed by clinical decision support (CDS)–guided RN visits. The intervention was implemented in February 2022 as a stepped-wedge trial in primary and urgent care practices within 4 academic health systems in New York, Wisconsin, and Utah [ 26 ].

Despite a seemingly straightforward URI clinical workflow, the RN-driven iCPR intervention encountered significant barriers early on during implementation. While these issues were pragmatically addressed as they arose during study implementation, a systematic assessment of the barriers to implementation is needed to better understand and address these barriers. The CFIR (Consolidated Framework for Implementation Research) [ 27 ] has been widely used to guide the systematic assessment of multilevel implementation contexts to identify contextual determinants of implementation success [ 28 ]. Using the updated CFIR as a guide [ 29 ], we sought to identify and categorize the barriers experienced during the implementation of the RN-driven antibiotic stewardship “iCPR3” intervention.

We performed a retrospective case study, collecting quantitative and qualitative data from expert interviews, study surveys, routine check-ins with practice personnel, and chart reviews over the first year of implementation of the iCPR3 intervention. We used the updated CFIR [ 29 ] to characterize the initial barriers to adapting a URI iCPR intervention for RNs in ambulatory care.

Ethical Considerations

The study protocol and procedures were approved by the NYU Langone Health institutional review board, which served as the study’s single institutional review board (NYULH Study: i19-01222). Informed consent was received from all participants. Documentation of consent was waived for this study. All study data reported in this manuscript are deidentified. Compensation was not provided for participation.

Study Intervention

The study intervention consists of triage followed by an in-person iCPR–guided RN visit for patients with low-acuity URIs ( Figure 1 ). RNs perform telephone triage (or in urgent care, an RN or medical assistant performs a similar assessment through a rooming protocol) for patients reporting cough or sore throat symptoms to assess acuity, need for primary care, urgent care, or ED visit, and appropriateness for an in-person iCPR–guided RN visit. In the urgent care setting, the assessment is dichotomous as either a need for a provider visit or appropriateness for an iCPR-guided RN visit. The triage tool consists of a prepopulated note template integrated into the EHR system designed to document patient symptoms and their severity and determine the most appropriate level of care. Triage algorithms were based on institutional triage resources for decisions about ED or urgent care visits, primary care visits, and home care [ 30 ].

Patients triaged as low acuity and appropriate for an RN visit are invited for an in-person RN visit that replaces the standard of care provider visit. During the RN visit, guided by iCPR tools, the RN evaluates a patient to determine their risk of bacterial infections of strep pharyngitis (sore throat) or pneumonia (cough). Prepopulated note template EHR tools lead RNs through a focused history and physical examination. Once an RN completes the patient history and physical examination, they use an iCPR tool specific to cough or sore throat to calculate the risk of bacterial infection based on the patient’s vitals, symptoms, and pertinent history [ 26 ]. The iCPR tools are informed by the CPRs [ 17 , 18 , 31 ] used in the iCPR1 and iCPR2 studies [ 12 , 19 ], which were validated in prior studies among patients with acute respiratory illnesses [ 17 , 18 , 31 ]. The CPRs are integrated into the EHR, and upon completion of the calculator, the level of risk with an approximate probability of having either strep pharyngitis or pneumonia is displayed. After completion of the risk calculator, the RN is linked by the EHR to an order set specific to the level of risk, along with relevant patient education.

Setting and Participants

The iCPR3 intervention study was implemented in February 2022, as a randomized controlled stepped-wedge trial, in 43 primary and urgent care practices associated with 4 academic medical centers including 2 in New York, 1 in Wisconsin, and 1 in Utah. To be eligible for participation, a practice must include general internal medicine, family medicine, or urgent care practices. Furthermore, practices must have at least 1 RN full-time equivalent capable of performing triage within the EHR and in-person RN visits.

For this case study, purposive sampling was used to select experts with key knowledge and insight on study implementation from members of the research team and study practices. This sample included research study team members engaged in the implementation of the iCPR3 study (ie, research coordinators, research assistants, and investigators) and study practice personnel (ie, RNs, RN or practice managers, and providers) from each study site (academic medical center). At least 2 research study team members per site participated in semistructured interviews, with those experts determining which practice personnel to include in their data collection. All RNs participating in the iCPR3 intervention were included in study acceptability surveys and routine implementation check-ins with study staff.

Of note, approximately 8 months into iCPR3 implementation, 1 New York–based study site withdrew from the intervention study due to limited practice recruitment and insurmountable barriers to implementing the intervention. Interviews were still performed with site personnel and their comments are included in these analyses.

Data Collection

A semistructured interview guide containing questions based on the 5 domains of CFIR [ 27 , 29 ] was developed. The CFIR constructs supported the research team in defining topics for the interviews and ensured that all major domains in the framework that influence implementation were addressed. Interview questions did not explicitly name or ask participants to name the CFIR domains or constructs. The interviews were performed via in-depth email interviews [ 32 ], in which research study team interviewees were asked questions to identify which of the 48 CFIR constructs were perceived as current barriers to iCPR3 implementation and provided detailed descriptions of the identified barriers and strategies that have already been used by the iCPR3 research study team.

Surveys and Routine Check-Ins

The perspectives of practice personnel were incorporated into the case study based on notes from surveys; individual interviews; or written feedback from RNs, providers, and RN and practice managers collected over the implementation period as routine intervention study procedures. As this was a pragmatic study, study staff routinely elicited informal feedback from practice personnel throughout intervention implementation to identify barriers and improve intervention implementation.

At 6 and 12 months post-RN visit implementation, participating RNs completed a short survey that asked about burnout, job satisfaction, and comfort levels with performing tasks related to treating patients reporting cough and sore throat. The survey also collected information on ease of use of the EHR tools as well as feedback on elements of the intervention, such as training, and recommendations.

Chart Review

Clinical workflows and EHR note templates (triage and RN visits) use in the first 12 months of implementation were collected via chart review. A subset of EHR template uses initiated was evaluated for appropriateness and completeness. To determine the total number of potential patients in a practice eligible for triage template use, patients with visits resulting in a diagnosis code for cough or sore throat were documented ( International Classification of Diseases-10 [ ICD-10 ] codes: R05, R07.0, J20.9, J06.9, and J18.9). The EHR records related to the visit were reviewed to determine patient eligibility for triage and document the workflow leading to the patient visit (ie, how the appointment was scheduled, by whom, and whether appointment notes were present).

CFIR Domains and Constructs

The CFIR was used to retrospectively describe the implementation process of the iCPR3 intervention to identify determinants in this process. Only the determinants relevant to the iCPR3 intervention implementation process were described. The CFIR is composed of 48 constructs sorted into 5 major domains including innovation, outer setting, inner setting, individuals, and implementation process [ 27 , 29 ]. Operationalization of CFIR domains for this study are shown in Table 1 .

a iCPR3: integrated clinical prediction rule 3.

b RN: registered nurse.

Data Coding and Analysis

Insights gathered from the surveys, chart reviews, and formal and informal check-ins with study practice personnel helped inform research study team members’ responses to the semistructured interview guide. The written responses and notes collected from the email interviews were analyzed using techniques of qualitative content analysis, inspired by a deductive-directed approach, deemed applicable because the data were analyzed in light of an existing framework [ 33 ]. The analysis was performed by 3 authors (ERS, LX, and JK) in a stepwise interactive process. The first step in the analysis, after reading all transcripts, notes, and written responses to obtain an understanding of the whole, was to develop initial coding nodes and subnodes based on the domains and constructs of the CFIR [ 29 ].

In the second step, units of analysis, such as sentences or sections of thought, were deductively coded into the nodes and subnodes. Third, the coded text was rated based on the recommended method described by the authors of CFIR, Damschroder and Lowery [ 34 ]. In the rating process, a consensus process was used to assign a rating to each construct obtained from each study site. The ratings reflected the positive or negative influence and the strength of each construct that emerged based on the coded text. When all constructs obtained from all study sites were rated, we compared and compiled ratings for each construct across study sites. Constructs were coded as missing, not distinguishing between positive or negative implementation factors (0), or weakly (+1/–1), or strongly (+2/–2) distinguishing low from high implementation factors ( Table 2 ).

a CFIR: Consolidated Framework for Implementation Research.

b iCPR3: integrated clinical prediction rule intervention.

Barriers and facilitators to implementation were identified within the CFIR domains and constructs and are presented within the frame of CFIR domains including innovation, outer and inner settings, individuals, and implementation process ( Table 3 ).

c Construct lettering and numbers correspond with Damschroder et al [ 27 ].

d Only constructs applicable to the iCPR implementation are cited.

e –2: strong negative influence; –1: weak negative influence; 0: neutral influence; 0 (mix): mixed positive and negative influences, which balanced each other; +1: weak positive influence; +2: strong positive influence; missing: not asked or miscoded.

Outer Setting

Local conditions , primarily driven by the COVID-19 pandemic, served as one of the strongest barriers to implementation as COVID-19 impacted nearly every aspect of implementation from changes in workflows and staffing availability to patient volume and URI care protocols. There were observed changes to URI care protocols including shifts from in-office care to telehealth and redirection to urgent care, driven by COVID-19–testing requirements and hesitancy from both patients and practices to have on-site care. Furthermore, COVID-19 affected local attitudes among practice staff as health issues and burnout led to staff shortages, turnover, and shifting of responsibilities. These barriers were further compounded by regional nursing shortages and financial incentives that drew RNs out of primary care practices.

Policies and laws, such as state regulatory laws and institutional policies, also had a strong impact on the study procedures and implementation. RN scope-of-practice varied by state and between institutions. Wisconsin has existing RN delegation protocols, allowing for more clinical autonomy among RNs than at institutions in New York and Utah. This required additional training and modification to the RN visit portion of the intervention at institutions, where RNs had a more limited scope of practice and could not function autonomously. For example, the New York sites were required to adopt a “co-visit” structure to ensure that providers could oversee RN visits. This created additional scheduling constraints and complexity, as well as an unanticipated burden for providers. Multimedia Appendix 1 shows the analysis of performance measurement pressure and the innovation construct.

Inner Setting

Within the construct of structural characteristics , work infrastructure served as a strong barrier to the intervention implementation as practices across institutions experienced staff changes, RN shortages and turnover, and competing responsibilities that all hindered their ability to effectively participate in the study. Notably, at practices with only 1 RN, implementation was negatively impacted as clinic participation was dependent on 1 individual, whereas at other practices, study responsibilities were distributed across multiple RNs. Within the culture construct, a norm of limited deliverer centeredness , related to the prioritization of the needs and desires of RNs, served as a barrier to the implementation of this RN-focused intervention. As patient ( recipient centeredness ) and provider preferences were prioritized over RN activities, the innovation activities that would have been performed by the RNs were overridden. For example, to ensure patient autonomy, if a patient preferred to see a provider, they were not scheduled for an RN visit even if they were eligible. Similarly, at most institutions (except those with more RN autonomy), RNs tended to defer to providers in terms of preference and final decisions. Therefore, if the provider preferred seeing a patient themselves, the patient, even if eligible for an RN visit, would not be seen by an RN.

Overall, relational connections , specifically the RN-provider dynamic, negatively affected implementation. RNs in the study did not always have open bidirectional communication with providers, thus limiting the self-efficacy of RNs to explain or justify intervention-related activities. As observed within the culture construct, many practices had limited deliverer centeredness , typically deferring to providers to make final decisions, and therefore RNs were hesitant to push these boundaries or make decisions that were contrary to a provider’s preferences. In particular, some sites mentioned some practices having poor relationships among practice staff, even requiring team-building training in some instances. On the other hand, this was less of a barrier at practices, where RNs had more clinical autonomy or had developed stronger relationships within the practice.

Communications culture within practices served as a barrier to effectively implementing aspects of the study; for example, some practices did not have a culture of communicating with patients prior to visits in the form of triage or lacked formalized documentation as information was often conveyed informally (eg, verbal, secure chat message, and free-text note). In some practices, a strong communication system between RNs (ie, a chat channel used by most RNs) served as a facilitator to innovation implementation by allowing RNs to support and answer each other’s questions.

The intervention’s compatibility , or lack thereof, with existing workflows was a strong barrier to implementation, as the necessary intervention-specific workflow adaptions required great effort on the part of the practice if not already in place (eg, front desk forwarding eligible patients for triage, RNs performing triage after appointments had been scheduled, and filling out EHR note templates as opposed to free text). As the new study workflow required changes to the status quo, tension for change also served as a barrier since practices perceived little anticipated benefit from the study as compared to the difficulty of change. Relatedly, relative priority of the intervention was a strong barrier as competing clinical responsibilities and the voluntary nature of the study meant staff would not prioritize study-related tasks.

Overall, there was a lack of incentive systems in place related to study activities, which hindered RN participation. While gift card incentives for RNs performing triage were used, these tended to incentivize the same RNs already using the tools as opposed to encouraging new RNs to participate. Additionally, at institutions where RNs were unable to bill for visits and did not receive any other recognition for their efforts, this lack of incentives was a strong barrier to participation. One institution was able to reduce the influence of this barrier by providing incentives to RNs through continuing education credits, an employee recognition fund, and paid time for training.

Multimedia Appendix 1 presents the analyses of physical infrastructure, IT infrastructure, access to knowledge and information, available resources, learning-centeredness, and mission alignment.

Individuals: Characteristics Subdomain

Both capability and motivation were barriers to implementation. As these tools were new to many of the participating RNs, they were less confident in their skills and required continuous feedback, training, and support. In addition, RNs were not motivated to participate in the study largely due to competing priorities, lack of a strong incentive, and COVID-19–related stress and burnout. Opportunity was also a strong barrier, as RNs did not have many opportunities to use the innovation tools. Conflicting responsibilities, staff shortages, workflow barriers, patient volume, and patient eligibility were observed as contributors to this barrier.

Multimedia Appendix 1 shows the analyses of roles subdomain constructs high-level leaders, mid-level leaders, opinion leaders, innovation deliverers, innovation recipients, implementation facilitators, implementation leaders, and implementation team members. Multimedia Appendix 1 shows the analyses of the implementation process domain constructs assessing context and assessing needs, innovation deliverers, doing, planning and tailoring strategies, teaming, engaging the innovation deliverers, reflecting and evaluating, and adapting.

Principal Findings

This case study identified numerous barriers to the successful implementation of iCPR3, an RN-driven antibiotic stewardship intervention. Many of the identified barriers are consistent with those observed in other interventions that sought to alter nursing responsibilities and workflows within primary care [ 35 , 36 ]. The most impactful barriers were noted within the outer setting, and these conditions were observed to influence the inner setting constructs. The effects of COVID-19 served as an overarching barrier that impacted nearly all implementation constructs, shifting the culture and conditions at many participating practices as well as decreasing the capacity of practices to engage in activities perceived as nonessential. These barriers, however, were less prevalent within clinics that had previously established workflows with patient care within the RN role description. Takeaways from this study can be applied to support integration and improve uptake during the implementation of other RN delegation protocols involving CDS tools into existing workflows.

Policies impacting innovation deliverers’ (RNs) clinical autonomy at both the state and institutional levels need to be considered when developing RN delegation protocols as they can impact implementation depending on compatibility with existing workflows. As a multisite study with implementation spanning 3 states, the differing state regulatory laws and institutional policies dictating RN scope-of-practice had a substantial impact on the compatibility of the iCPR3 implementation at each site. This was evident in the higher rate of RN visits occurring in practices in Wisconsin compared to New York. At the Wisconsin study site, there were established delegation protocols for RNs to see patients with minimal provider supervision. In contrast, for the 2 New York study sites, a more complex “covisit” design was developed, which involved joint scheduling of the iCPR3 RN visit followed immediately by a visit for the provider to see the patient and confirm the RN plan of care. The addition of a provider visit component increased the intervention’s dependency on already limited provider availability, thus inhibiting the ability to schedule the iCPR3 RN visits even when a patient was appropriate and willing and an RN was available to conduct the visit. As observed in other RN delegation protocols, considerations for local regulations must be made when assessing the viability of implementing these types of interventions [ 36 ].

Consideration of practice-level culture and work infrastructure is also essential for the successful implementation of an intervention that includes RN delegation protocols. This implementation study revealed impactful differences in existing workflow expectations that affected RN capability and intervention complexity. One unexpected barrier was the influence of practice personnel who were part of the local workflow but were not directly involved in the implementation of the iCPR tools. For example, at one institution, successful implementation of the intervention was reliant on administrative staff to forward patients reporting cough and sore throat to participating RNs for triage. Implementation planning with greater efforts to clarify practice-level workflows, identifying potential stakeholders early on, and engaging these personnel who ultimately support the innovation deliverers can support a successful implementation.

Similarly, when delegating provider tasks to RNs, it is important to secure provider buy-in early on in the implementation process, even with a seemingly RN-focused intervention. Consistent with previous research demonstrating the importance of RN-provider relationships in job satisfaction [ 37 ], this study showed that power dynamics between providers and RNs can serve as a barrier to RN intervention engagement. With a culture of deference to providers, many RNs did not want to overstep these boundaries and would not engage with the intervention if there was any perceived resistance from practice providers. Barriers experienced due to this power structure were further compounded when poor relations existed between RNs and providers. Furthermore, as seen in other clinical academic partnerships, future implementation efforts would benefit from more active engagement of leadership at all levels [ 38 ].

Future clinical delegation interventions may also need to consider alternate care mechanisms to account for unexpected shifts in clinic workflows. Due to the timing of the implementation, one of the largest observed barriers to implementation was the COVID-19 pandemic, which amplified nearly all other barriers and created additional unique challenges. As an intervention specifically designed for in-person care, the shift toward telemedicine driven by the pandemic [ 39 ] had a particularly negative impact on implementation. One study institution piloted a program to divert all patients with URI to telemedicine visits with a centrally employed nurse practitioner, which bypassed all potential points of intervention for the iCPR study. Further diverting potentially eligible patients away from primary care practices was the increased popularity of urgent care centers [ 40 ], which served as an expedient solution for patients with URI seeking to avoid long wait times at many primary care practices. Incorporating alternate care mechanisms to provide agility in the intervention may support the success of study implementation. Similarly, integrating CDS tools with existing EHR tools and templates can help minimize changes in workflow, thereby allowing interventions to be resilient in the face of unforeseen events.

As observed in this case study, the pandemic also directly impacted practice staff and their ability to participate in activities beyond the essential, including research. Practices across all study institutions experienced nursing staff shortages due to RNs themselves being sick, covering for others who are sick, or leaving the practice altogether, thus resulting in a redistribution of responsibilities. An increased workload, along with outside stressors, led to increased reported stress and burnout among practice staff [ 41 ], making it difficult for them to view the study as a daily priority. The voluntary nature of the study and these conflicting responsibilities greatly reduced the opportunity for RNs to use the innovation and participate fully. This was particularly evident in practices that required greater workflow modifications. Practices with existing expectations of note documentation and template use facilitated implementation; however, in other practices, the lack of RN familiarity with these EHR functions required the creation of additional training and workflow modification efforts, as well as a greater perceived effort burden on the part of RNs.

Future implementation should consider the value of face-to-face communication in encouraging engagement and team building during the implementation process [ 42 ]. In addition to its impact at the institutional and practice level, the effects of COVID-19 hindered the implementation process itself, especially early in the planning phase by limiting in-person interactions and creating communication barriers [ 43 ]. With nearly all communication occurring remotely, interactions to collect practice workflow information and engage stakeholders were perceived as less efficient, requiring additional follow-up meetings and hindering the development of relationships of the study team with leadership and innovation deliverers. When in-person practice visits by the research team became feasible, an improvement in practice responsiveness and innovation uptake was observed [ 42 ].

This study had several limitations. First, the use of an emailed in-depth interview hindered the study team’s ability to probe respondents for further information at the moment, potentially limiting the collection of further details that may have impacted the interpretation of interview responses. However, the emailed format increased the feasibility of conducting a long interview and created an opportunity for study sites to compile perspectives from multiple team members, thus improving the richness of information provided. Second, the reported barriers and facilitators were self-reported and not directly observed and are therefore based on the perceptions of the study site research teams. Similarly, as the data collection was primarily retrospective, it may be subject to recall bias. We attempted to mitigate this by conducting semistructured interviews during the implementation process. Finally, this analysis was performed prior to the completion of implementation at all sites and analysis of the primary intervention effectiveness outcomes. Therefore, it was not possible to link perceived implementation constructs to intervention outcome measures, and additional implementation construct influences may have been missed.

Conclusions

Both between and within health care systems, significant variability exists in workflows for patient intake and triage. Even in a relatively straightforward clinical workflow, seemingly nuanced workflow and culture differences appreciably impacted successful intervention adoption. Barriers to intervention adoption existed within multiple constructs and domains. When implementing a system-wide clinical care intervention, stakeholders should consider the variability in workflow policy and culture at the health system, practice, and individual levels, as well as create accommodations for changing care patterns.

Acknowledgments

This work was funded by NIAID 5R01AI108680 (PI Mann). Generative AI was not used in the writing of this manuscript.

Data Availability

The data sets used and analyzed during this study contain personally identifiable information and are therefore not made publicly available. Data are available from the corresponding author upon reasonable request.

Authors' Contributions

ERS wrote the original draft and contributed to conceptualization, methodology, data collection, and analysis; LX contributed to writing the original draft, methodology, data collection, and analysis; JK contributed to writing the original draft, methodology, data collection, and analysis; ST contributed to data collection and project administration; NH contributed to data collection and writing editing; DF contributed to data collection and writing editing; EJK contributed to data collection and writing editing; RH contributed to data collection and writing editing; KLD-D contributed to data collection and writing editing; PDS contributed to data collection and writing editing; WH contributed to data collection and writing editing; PG-G contributed to data collection and writing editing; DAF contributed to the conceptualization, funding acquisition, and writing editing; DMM contributed to the conceptualization, funding acquisition, and writing editing. All authors reviewed and approved the final manuscript draft.

Conflicts of Interest

None declared.

Consolidated Framework for Implementation Research domain barrier analysis results.

• Antibiotic resistance threats in the United States, 2019. Centers for Disease Control and Prevention. 2019. URL: https://www.hhs.gov/sites/default/files/michael-craig-cdc-talk-thursday-am-508.pdf [accessed 2024-04-10]
• Dadgostar P. Antimicrobial resistance: implications and costs. Infect Drug Resist. 2019;12:3903-3910. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bush K, Courvalin P, Dantas G, Davies J, Eisenstein B, Huovinen P, et al. Tackling antibiotic resistance. Nat Rev Microbiol. 2011;9(12):894-896. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nadimpalli ML, Chan CW, Doron S. Antibiotic resistance: a call to action to prevent the next epidemic of inequality. Nat Med. 2021;27(2):187-188. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fleming-Dutra KE, Hersh AL, Shapiro DJ, Bartoces M, Enns EA, File TM, et al. Prevalence of inappropriate antibiotic prescriptions among US ambulatory care visits, 2010-2011. JAMA. 2016;315(17):1864-1873. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Havers F, Thaker S, Clippard JR, Jackson M, McLean HQ, Gaglani M, et al. Use of influenza antiviral agents by ambulatory care clinicians during the 2012-2013 influenza season. Clin Infect Dis. 2014;59(6):774-782. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Duffy E, Ritchie S, Metcalfe S, Van Bakel B, Thomas MG. Antibacterials dispensed in the community comprise 85%-95% of total human antibacterial consumption. J Clin Pharm Ther. 2018;43(1):59-64. [ CrossRef ] [ Medline ]
• Consumption of antibiotics and occurrence of antibiotic resistance in Sweden. Swedres-Svarm. URL: https://www.sva.se/media/jzdlctnk/rapport_swedres-svarm_2018.pdf [accessed 2024-04-10]
• Hopkins S, Muller-Pebody B, Guy R, Gerver S, Ironmonger D, Puleston R, et al. English Surveillance Programme for Antimicrobial Utilisation and Resistance (ESPAUR) 2010 to 2014: Report 2015. UK National Archives. 2015. URL: https:/​/webarchive.​nationalarchives.gov.uk/​ukgwa/​20181130125613/​https:/​/www.​gov.uk/​government/​publications/​english-survei llance-programme-antimicrobial-utilisation-and-resistance-espaur-report [accessed 2024-04-10]
• Barnett ML, Linder JA. Antibiotic prescribing for adults with acute bronchitis in the United States, 1996-2010. JAMA. 2014;311(19):2020-2022. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Barnett ML, Linder JA. Antibiotic prescribing to adults with sore throat in the United States, 1997-2010. JAMA Intern Med. 2014;174(1):138-140. [ FREE Full text ] [ CrossRef ] [ Medline ]
• McGinn TG, McCullagh L, Kannry J, Knaus M, Sofianou A, Wisnivesky JP, et al. Efficacy of an evidence-based clinical decision support in primary care practices: a randomized clinical trial. JAMA Intern Med. 2013;173(17):1584-1591. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Linder JA, Schnipper JL, Tsurikova R, Yu T, Volk LA, Melnikas AJ, et al. Documentation-based clinical decision support to improve antibiotic prescribing for acute respiratory infections in primary care: a cluster randomised controlled trial. Inform Prim Care. 2009;17(4):231-240. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gonzales R, Anderer T, McCulloch CE, Maselli JH, Bloom FJ, Graf TR, et al. A cluster randomized trial of decision support strategies for reducing antibiotic use in acute bronchitis. JAMA Intern Med. 2013;173(4):267-273. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Keller SC, Tamma PD, Cosgrove SE, Miller MA, Sateia H, Szymczak J, et al. Ambulatory antibiotic stewardship through a human factors engineering approach: a systematic review. J Am Board Fam Med. 2018;31(3):417-430. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fine AM, Nizet V, Mandl KD. Large-scale validation of the centor and McIsaac scores to predict group a streptococcal pharyngitis. Arch Intern Med. 2012;172(11):847-852. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Heckerling PS, Tape TG, Wigton RS, Hissong KK, Leikin JB, Ornato JP, et al. Clinical prediction rule for pulmonary infiltrates. Ann Intern Med. 1990;113(9):664-670. [ CrossRef ] [ Medline ]
• McIsaac WJ, White D, Tannenbaum D, Low DE. A clinical score to reduce unnecessary antibiotic use in patients with sore throat. CMAJ. 1998;158(1):75-83. [ FREE Full text ] [ Medline ]
• Mann D, Hess R, McGinn T, Richardson S, Jones S, Palmisano J, et al. Impact of clinical decision support on antibiotic prescribing for acute respiratory infections: a cluster randomized implementation trial. J Gen Intern Med. 2020;35(Suppl 2):788-795. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zetts RM, Garcia AM, Doctor JN, Gerber JS, Linder JA, Hyun DY. Primary care physicians' attitudes and perceptions towards antibiotic resistance and antibiotic stewardship: a national survey. Open Forum Infect Dis. 2020;7(7):ofaa244. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Phansalkar S, van der Sijs H, Tucker AD, Desai AA, Bell DS, Teich JM, et al. Drug-drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc. 2013;20(3):489-493. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R, et al. HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak. 2017;17(1):36. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Shaw RJ, McDuffie JR, Hendrix CC, Edie A, Lindsey-Davis L, Williams JW. Effects of nurse-managed protocols in the outpatient management of adults with chronic conditions. Department of Veterans Affairs. 2013. URL: https:/​/www.​care innovations.org/​wp-content/​uploads/​Nurse-Managed-Protocols-in-Adults-with-Chronic-Disease_VA.​pdf [accessed 2024-04-10]
• Iglesias B, Ramos F, Serrano B, Fàbregas M, Sánchez C, García MJ, et al. A randomized controlled trial of nurses vs. doctors in the resolution of acute disease of low complexity in primary care. J Adv Nurs. 2013;69(11):2446-2457. [ CrossRef ] [ Medline ]
• Shum C, Humphreys A, Wheeler D, Cochrane MA, Skoda S, Clement S. Nurse management of patients with minor illnesses in general practice: multicentre, randomised controlled trial. BMJ. 2000;320(7241):1038-1043. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stevens ER, Agbakoba R, Mann DM, Hess R, Richardson SI, McGinn T, et al. Reducing prescribing of antibiotics for acute respiratory infections using a frontline nurse-led EHR-integrated clinical decision support tool: protocol for a stepped wedge randomized control trial. BMC Med Inform Decis Mak. 2023;23(1):260. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kirk MA, Kelley C, Yankey N, Birken SA, Abadie B, Damschroder L. A systematic review of the use of the consolidated framework for implementation research. Implement Sci. 2016;11:72. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci. 2022;17(1):75. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Briggs JK. Telephone Triage Protocols for Nurses, Fifth Edition. Philadelphia, PA. Wolters Kluwer; 2015.
• Centor RM, Witherspoon JM, Dalton HP, Brody CE, Link K. The diagnosis of strep throat in adults in the emergency room. Med Decis Making. 1981;1(3):239-246. [ CrossRef ] [ Medline ]
• Fritz RL, Vandermause R. Data collection via in-depth email interviewing: lessons from the field. Qual Health Res. 2018;28(10):1640-1649. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277-1288. [ CrossRef ] [ Medline ]
• Damschroder LJ, Lowery JC. Evaluation of a large-scale weight management program using the Consolidated Framework for Implementation Research (CFIR). Implement Sci. 2013;8:51. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Karimi-Shahanjarini A, Shakibazadeh E, Rashidian A, Hajimiri K, Glenton C, Noyes J, et al. Barriers and facilitators to the implementation of doctor-nurse substitution strategies in primary care: a qualitative evidence synthesis. Cochrane Database Syst Rev. 2019;4(4):CD010412. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Busca E, Savatteri A, Calafato TL, Mazzoleni B, Barisone M, Dal Molin A. Barriers and facilitators to the implementation of nurse's role in primary care settings: an integrative review. BMC Nurs. 2021;20(1):171. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Galletta M, Portoghese I, Carta MG, D'Aloja E, Campagna M. The effect of nurse-physician collaboration on job satisfaction, team commitment, and turnover intention in nurses. Res Nurs Health. 2016;39(5):375-385. [ CrossRef ] [ Medline ]
• Baptiste DL, Whalen M, Goodwin M. Approaches for establishing and sustaining clinical academic partnerships: a discursive review. J Clin Nurs. 2022;31(3-4):329-334. [ CrossRef ] [ Medline ]
• Mann DM, Chen J, Chunara R, Testa PA, Nov O. COVID-19 transforms health care through telemedicine: evidence from the field. J Am Med Inform Assoc. 2020;27(7):1132-1135. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ayers A. As COVID turns endemic, investors remain bullish on urgent care growth. J Urgent Care Med. 2022;16(10):35-39. [ FREE Full text ]
• Restauri N, Sheridan AD. Burnout and posttraumatic stress disorder in the coronavirus disease 2019 (COVID-19) pandemic: intersection, impact, and interventions. J Am Coll Radiol. 2020;17(7):921-926. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lederman O. Hacking innovation—group dynamics in innovation teams. MIT Libraries. 2015. URL: https://dspace.mit.edu/handle/1721.1/101790 [accessed 2024-04-10]
• Mourad M, Bousleiman S, Wapner R, Gyamfi-Bannerman C. Conducting research during the COVID-19 pandemic. Semin Perinatol. 2020;44(7):151287. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Mavragani; submitted 30.11.23; peer-reviewed by R Hillard; comments to author 21.02.24; revised version received 13.03.24; accepted 15.03.24; published 23.05.24.

©Elizabeth R Stevens, Lynn Xu, JaeEun Kwon, Sumaiya Tasneem, Natalie Henning, Dawn Feldthouse, Eun Ji Kim, Rachel Hess, Katherine L Dauber-Decker, Paul D Smith, Wendy Halm, Pranisha Gautam-Goyal, David A Feldstein, Devin M Mann. Originally published in JMIR Formative Research (https://formative.jmir.org), 23.05.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.

Blockchain technology implementation challenges in supply chains – evidence from the case studies of multi-stakeholders

The International Journal of Logistics Management

ISSN : 0957-4093

Article publication date: 2 December 2022

Issue publication date: 19 December 2022

The aim of the research is to identify and prioritise the implementation challenges of blockchain technology and suggests ways for its implementation in supply chains.

Design/methodology/approach

Underlined by the technology, organisational, and external environment model, a conceptual framework with four challenge categories and sixteen challenges is proposed. Data collected from three stakeholder groups with experience in the implementation of blockchain technology in India is analysed by employing an analytical hierarchy process method-based case study. Further, a criticality–effort matrix analysis is performed to group challenges and suggest ways for implementation.

The analysis revels that all stakeholders perceive complexity challenge associated with the technology, organisational structure, and external environment, and issues of compatibility with existing systems, software, and business practices to be high on the criticality and effort scales, which thus require meticulous planning to manage. Likewise, top-management support issues related to insufficient understanding of how technology fits with the organisation’s policy and benefits offered by the technology requires high effort to address this challenge.

Research limitations/implications

The results were obtained by focusing on the Indian context and therefore may not apply to other nations’ contexts.

Practical implications

By investigating the challenges that the developers, consultants, and client organisations need to address, this study assists managers in developing plans to facilitate coordination among these organisations for successful blockchain implementation.

Originality/value

To the authors’ knowledge this study is the first to identify and prioritise the challenges from the perspectives of multiple stakeholder groups with experience in blockchain technology implementation.

• Analytical hierarchy process (AHP)
• Organisation
• And environment framework (TOE)
• Supply chain management

Yadlapalli, A. , Rahman, S. and Gopal, P. (2022), "Blockchain technology implementation challenges in supply chains – evidence from the case studies of multi-stakeholders", The International Journal of Logistics Management , Vol. 33 No. 5, pp. 278-305. https://doi.org/10.1108/IJLM-02-2021-0086

Emerald Publishing Limited

Copyright © 2020, Aswini Yadlapalli, Shams Rahman and Pinapala Gopal

Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode

1. Introduction

Blockchain refers to a ledger of data transactions recorded on a distributed database and shared with a network of independent participants ( Perdana et al. , 2021 ). Because data are recorded on a decentralised system that participants cannot control, it ensures that no one owns the system ( Upadhyay, 2020 ). Blockchain data records are referred to as blocks, and these are connected in a chain using the crypto-analytic hash function. The hash function used to validate the transactions on blocks prevents alterations to recorded data ( Wang et al. , 2019a ). The ability of a block to remain unchanged and unaltered is referred to as immutability. The decentralisation and immutability properties of the technology can revolutionise business operations because these allow the sharing of agreed information that cannot be altered by partner organisations in supply chains ( Gurtu and Johny, 2019 ; Kshetri, 2021 ). These characteristics of the technology can extend its application to building trusting relationships between the organisations in supply chains ( Queiroz and Wamba, 2019 ).

Further, the smart contract feature of blockchain technology is a computer protocol that facilitates the verification and enforcement of the negotiated terms of a contract without the need for third-party intervention ( Upadhyay, 2020 ). The protocols of a smart contract can validate various transactions, such as payment processing or asset verification ( Cole et al. , 2019 ). By ensuring that all participants are obeying the rules, smart contracts instil confidence among supply chain members. Moreover, the recent development of a permissioned blockchain that limits data access to participants who have an invitation or the permission to join the network has offered much-needed data privacy to supply chain members ( Nandi et al. , 2020 ; Wong et al. , 2020b ). Besides, implementing blockchain technology in supply chains leads to improved efficiency in forecasting demand, managing inventory, tracing the product origin, and managing the supply chain finance process ( Hald and Kinra, 2019 ). Hence, organisations are increasingly seeking ways to adopt this technology in supply chains ( Perdana et al. , 2021 ).

Most well-known blockchain technology applications in supply chains are for the traceability of products ranging from essential food items to luxury diamonds, which is aimed at promoting consumer confidence ( Gurtu and Johny, 2019 ). Similarly, blockchain-based supply chains can detect counterfeits in product categories ranging from consumer products of less public concern to medicines, a serious public health concern that endangers lives ( Gaur, 2020 ). In the transport and logistics industry, using blockchain technology to digitise and transfer key trade documents, such as the bill of lading and customs documents, can improve process efficiency and increase global trade by 15% through minimising the barriers to trade ( DHL, 2019 ).

Despite the benefits of blockchain technology when implemented in supply chains, its implementation is confined to the proof-of-concept stage ( Cole et al. , 2019 ), with limited mainstream adoption in supply chains. In a global survey, 28% of executives rated a low level of understanding of this technology as a major barrier to its implementation ( Deloitte, 2019 ). Moreover, practitioners with some understanding indicated that the lack of knowledge on implementation factors hinders their technology uptake ( Gaur, 2020 ). Saberi et al. (2019) and Kouhizadeh et al. (2021) highlighted that the successful implementation of this technology begins with identifying related challenges. Therefore, to promote blockchain diffusion in supply chains, it is necessary to ascertain the value expected to be created and the implementation strategies required to materialise its value. Hence, a significant research area is an identification, prioritisation, and development of strategies for implementing blockchain technology.

Academic literature on blockchain implementation in supply chains has started appearing from 2015 with many conference papers at the start followed by few journal articles ( Wang et al. , 2019a ). Hald and Kinra (2019) , Saberi et al. (2019) , and Tokkozhina et al. (2022) identified blockchain technology implementation challenges through literature synthesis and highlighted the need for empirical research to examine these challenges. Accordingly, Queiroz and Wamba (2019) , Wang et al. (2019a , b) , Wong et al. (2020b) , and Kouhizadeh et al. (2021) investigated the blockchain implementation challenges in India, Europe, Indonesia, Malaysia, and United States. These studies have identified complexity, lack of financial resources, lack of management support, trust, and privacy as the blockchain implementation challenges. These studies use data collected from the logistics practitioners and academics who are familiar with the technology but lack experience in implementing it. Their lack of experience raises concern about their ability to evaluate the implementation of this technology. Also, the empirical research examining real cases to investigate blockchain implementation challenges is limited warranting future research. Our study is different from various perspectives, including the scope, the methodology, the survey sample and the analysis. Particularly, this study collects the responses from the multiple stakeholders who are not only knowledgeable but also involved in the blockchain implementation. We address the objective through analytics hierarchy process (AHP)-based case study using both qualitative and quantitative data collected from various stakeholders, such as consultants, developers, and clients, who are experienced in implementing blockchain technology. The case study approach used in this study is considered appropriate as it elaborates the understanding of challenges influencing blockchain technology implementation in supply chains in Indian context ( Shah and Corley, 2006 ; Ketokivi and Choi, 2014 ). Moreover, conducting a case study is regarded as a suitable method to explore the complex phenomena surrounding the implementation of disruptive blockchain technology ( Yin, 2003 ).

The present study contributes to the literature in two ways. First, this is the first such study to prioritise challenges in implementing blockchain technology in supply chains. Prioritising the identified challenges would assist in developing strategies that promote the implementation of blockchain technology. Second, the study compares the perspectives of multiple stakeholders who have adequate experience in implementing this technology. Such comparisons are critical for understanding and addressing the differences between stakeholder preferences as they join the consortium and collaborate for implementing blockchain technology.

The remainder of the paper is organised as follows. We propose a conceptual framework in section 2 through a review of the literature on technology implementation in supply chains. We discuss the research methodology to identify critical challenges in section 3 . We present the study results and related analysis in section 4 and discuss these results and their implications in section 5 and 6 respectively. We conclude the paper in section 7 by discussing the limitations of this study and future research areas.

2. Background

2.1 blockchain technology.

Based on the developments and their applications, the evolution of blockchain technology can be divided into three phases. In Phase 1, blockchain is mainly used as a cryptocurrency in applications related to currency transfer, remittance, and digital payments.

In Phase 2, businesses have realised decentralized ledger an underlying principle of blockchain technology can be separated from the cryptocurrency application and used for inter-organisational collaboration. The development of decentralised applications (DApp) is considered as the major aspect of blockchain evolution in Phase 2.

Advancement in Phase 3 of blockchain technology evolution can be seen in automating the validation process of data recording through the internet of Things (IoT) ecosystem. Integration of IoT into blockchain creates a “de facto standardized Ledger of Everything” that brings the highest degree of accountability with no more human errors and missed transactions ( Pournader et al. , 2020 ). Overall, blockchain technology has evolved from being used as a digital currency application towards a wider decentralised application with the ability of automation in transaction validation.

2.2 Literature on blockchain in supply chains

Limited academic literature available on blockchain technology in supply chains can be summarised into two categories. First category of papers examines the application of blockchain technology in supply chain domains, while the second category identifies the drivers and challenges of blockchain implementation in supply chains. We discuss these two categories of papers in the following sections.

2.2.1 Blockchain application in supply chain domains

The current application of blockchain technology in supply chains falls within three broad supply chain domains such as sourcing (buy), logistics, and finance A summary of these studies is provided in (see Table 1 ) and discussed below.

Buy (sourcing) function referred to as procurement plays an important role in identifying and managing the intra- and inter-organisational issues which impact supply chain resilience. The use of blockchain technology to trace the product origins assists in making sure the products are from conflict-free sources and thus promoting trust among the supply chain members ( Kshetri, 2021 ). Moreover, the distributor ledger concept behind the blockchain technology is much like a stock ledger with the information on the purchase orders, inventory levels, goods received, shipping manifests, and invoices that can be accessed by all the supply chain members instantaneously promoting data visibility among the members ( Cole et al. , 2019 ). Overall, the availability of accurate demand forecast information also assists in managing resources effectively and reduces inventory carrying costs which facilitate the implementation of process improvement tools such as lean and six-sigma in supply chains ( Kamble et al. , 2019 ).

Logistics assists in the management and coordination of freight transport, storage, inventory management, materials handling, and information processing activities. Greater dependency on logistics services for the distribution of products from sourcing to consumption through production in global supply chains has made the logistics industry to play a critical role in efficient supply chains ( Kamble et al. , 2019 ). For seamless information flows between logistics service providers and supply chain members, the resources used for the distribution of products such as vehicles and handling equipment should be integrated with technologies such as GPS, sensors, IoT devices, or automatic image-recognition software that provides the live information to blockchain distributed database ( Vivaldini, 2021 ). Once such integration has been achieved, the permanent nature of blockchain will ensure that data cannot be modified at any time in the future. Moreover, the technology also enhances the customer experience by enabling them to trace and track the product live ( Wang et al. , 2019a ).

Supply chain finance became crucial after the global financial crisis due to the less credit availability and higher borrowing costs. To optimise financial flows in supply chains, organisations are aligning financial flows with product and information flows through technology. Smart contracts of blockchain technology facilitate supply chain finance through matching and verifying the recorded data against the agreement and trigger payment which may or may not be in bitcoin or another cryptocurrency ( Babich and Hilary, 2020 ). It can autonomously trigger other transactions when key milestones are met, such as goods being issued (creating a shipment), pickup confirmed (activating a sensor), or proof of delivery (issuing an invoice). The automation of initiating purchase orders or invoices without the use of spreadsheets or manual interference speed up the transactions and minimises the costs and time associated with intermediation ( Cole et al. , 2019 ).

In spite of the benefits, blockchain technology adoption in supply chains is relatively slow and very much limited to pilot studies ( Kouhizadeh et al. , 2021 ). Identifying and addressing the challenges that impede blockchain implementation became an important topic for investigation ( Caldarelli et al. , 2021 ). Following provides the literature on drivers and challenges of blockchain implementation in supply chains.

2.2.2 Drivers and challenges of blockchain implementation in supply chains

Literature on blockchain implementation in supply chains can be classified into two streams. First stream of literature focuses on investigating the factors driving the implementation of blockchain technology in supply chain (see for example: Kamble et al. , 2019 ; Queiroz and Wamba, 2019 ; Wong et al. , 2020a ). Kamble et al. (2019) identified the perceived usefulness of the technology and attitude of the users as the factors affecting the intention to implement blockchain technology among supply chains operating in India. Meanwhile, Queiroz and Wamba (2019) study highlighted distinct blockchain adoption behaviours between India-based and USA-based professionals. Wong et al. (2020a) identified facilitating conditions, technology readiness of the firm, and technology affinity as the factors influencing the managerial intention to implement blockchain technology among the SMEs in Malaysia. In the context of Brazil, Queiroz et al. (2021) recognised effort expectancy, facilitating conditions, trust and social influence as the factors impacting the intention to implement blockchain technology in supply chains.

Second stream of literature emphasises on examining the challenges impacting the blockchain implementation in supply chains. Casey and Wong (2017) highlighted the interoperability between different blockchains and the complexity of the rules and regulations that govern the implementation as the challenges impacting the blockchain implementation in supply chains. Through interviewing supply chain experts from multiple countries, Wang et al. (2019b) reported that complexity of the technology, high cost of implementation, lack of clear governance rules, and interoperability between two or more different blockchains and compatibility with other existing systems as the challenges of blockchain implementation. Wong et al. (2020b) in the context of Malaysia identified the pressure from competition in the market, complexity, financial resources, and relative sustainable advantage impact the implementation of blockchain technology. Meanwhile, Kouhizadeh et al. (2021) recognised lack of management commitment and support, lack of knowledge and expertise, lack of cooperation, coordination and information disclosure between supply chain members, lack of policies and industry involvement as the barriers. More recently, Caldarelli et al. (2021) identified scalability, implementation costs, and lack of standards as the challenges of blockchain implementation in apparel supply chains.

3. Blockchain implementation challenges: a conceptual framework

To understand the implementation of blockchain technology in supply chains, it is important to examine the factors influencing implementation decision-making, which is the objective of this study. Different technology adoption models and theories, such as the technology acceptance model, the theory of planned behaviour, the theory of reasoned action, the unified theory of acceptance and use of technology, the diffusion of innovation (DOI) theory, and the technological, organisational and environmental (TOE) model are used to understand factors facilitating the implementation of the technology (e.g. Chong and Ooi, 2008 ; Lin, 2014 ). However, apart from the TOE framework and the DOI theory, all the other theories are individual-level theories that examine individual attitudes towards technology implementation. Therefore, they are not appropriate for examining technology implementation at the organisational level ( Bradford et al. , 2014 ).

In the supply chain context, the TOE framework has been applied to study the implementation of various internet-based supply chains management technologies, such as e-business, e-commerce, information and communications technology, enterprise resource planning (ERP), electronic data interchange (EDI), radio frequency identification (RFID) and cloud computing ( Low et al. , 2011 ; Chan et al. , 2012 ). More recently, researchers have used the TOE framework to examine the factors impacting blockchain technology adoption ( Saberi et al. , 2019 ; Caldarelli et al. , 2021 ; Kouhizadeh et al. , 2021 ). In line with the previous studies, the TOE framework is used as an underlying theory in this study.

The TOE framework presents the technology, the organisation, and the external environment as the three factors that influence firms’ decision-making of adopting and implementing innovations. Traditional TOE models have focused at the organisational level and excluded inter-organisational relationship aspects such as the position of the firm in supply chains, trust amongst the supply chain partners, and collaboration between the firms ( Chan et al. , 2012 ). Chong and Ooi (2008) have identified the inter-organisational relationship as a crucial factor influencing the technology adoption between organisations. Table 2 presents the literature using the TOE framework to investigate technology adoptions in supply chains. In the context of blockchain technology in supply chains, challenges related to interorganisational aspects need to be addressed for successful technology implementation ( Saberi et al. , 2019 ). The theory elaboration approach considered in this study allows to add more variables to the existing framework ( Ketokivi and Choi, 2014 ). Similar to Chong and Ooi (2008) , Huang et al. (2008) , Chan et al. (2012) , and Kouhizadeh et al. (2021) this study considers the technology, organisational, external environment and interorganisational categories as the challenge categories of blockchain technology implementation in the supply chains.

3.1 Technology challenge category

A review of the recent adoption models reveals that the Roger’s (1995) DOI theory is used to identify the technology-related factors affecting the innovation adoption rate. In our study, DOI is used to provide a theoretical explanation of the technological challenges of the TOE framework ( Baker, 2011 ). The DOI theory defines compatibility, complexity, relative advantage, trialability, and observability as the factors affecting technological implementation ( Rogers, 1995 ). In the context of blockchain technology, current advancements will not be able to replace the existing systems, and therefore, a blockchain-based system should be compatible with the existing legacy system ( Lielacher, 2018 ). The technical interfaces used to connect the two systems add complexity to the implementation process. Moreover, the lack of earlier full-scale adoption of the technology in supply chains obstructs the firm’s ability to ensure its successful implementation and will impede it in realising the relative advantages offered by the technology in comparison with the conventional centralised database structure. Besides, flexibility to trial the technology in the supply chain process will play an important role in its implementation.

3.2 Organisational challenge category

Organisational context in this study refers to several factors, such as top-management support, technical know-how, financial resources, and firm size, which facilitate technology adoption in organisations ( Tornatzky and Fleischer, 1990 ). In particular, organisations with a high degree of centralisation of power with the top management are likely to make adoption decisions irrespective of resistance from lower-level managers and employees. Regarding the organisation size, large organisations can invest in resources that facilitate implementation. However, the agility and the flexibility of smaller organisations facilitate their adoption of innovations ( Wang et al. , 2019a ). Further, because organisational resources, such as financial resources and technical expertise, influence decisions on technology implementation, it is important to understand their role in the implementation process. In particular, the newness of blockchain technology and the lack of readily available, off-the-shelf software may result in greater costs for organisations ( Lielacher, 2018 ).

3.3 External environment challenge category

External environment factors such as the industry structure, the security provided by the technology service provider, and the regulatory environment may become constraints or provide opportunities for technology implementation in supply chains ( Huang et al. , 2008 ; Bradford et al. , 2014 ). The lack of government regulations regarding the recording of transactions on the blockchain has bypassed the inefficiencies likely to result from following such regulations. Despite the lack of government standards to guide blockchain implementation, organisations are carefully seeking industry use cases to understand the industry characteristics influencing blockchain implementation. Moreover, the presence of external technology providers with more than 50% mining power to validate new transactions on blockchain raises a security concern regarding the data recorded on blockchain ( Lielacher, 2018 ).

3.4 Interorganisational relationships challenge category

An interorganisational relationship is a complex construct with many dimensions, such as the partner’s power, information sharing, privacy, and trust. According to Chong and Ooi (2008) , partner power is an organisation’s ability to exert influence on another company to act in a prescribed manner. Supply chain members who trust each other will achieve the benefits of technology implementation. Thus, blockchain technology implementation depends on the degree of trust between business partners ( Saberi et al. , 2019 ). Information sharing is crucial to a successfully integrated supply chain. Despite its importance, firms do not have the confidence to share information with the members of their supply chains because their competitors may obtain this information, which would affect the firms’ business ( Chan et al. , 2012 ). Hence, the privacy that this technology offers plays an important role in its diffusion in supply chains. Based on this discussion, we propose a conceptual framework of the challenges in implementing blockchain technology in supply chains (see Figure 1 ).

4. Research methodology

Using quantitative methods to analyse a case assists in theory elaboration by providing rich insights into the context ( Kaplan and Duchon, 1988 ). In this study, to identify the critical challenges from the proposed framework, case study methodology is adopted, and AHP is used to analyse the quantitative data of the interviews. In the context of technology adoption, AHP has been widely used to analyse the data. For example, the method was used to identify the challenges influencing the decision-making regarding technology adoption ( Bigdeli et al. , 2013 ), to investigate market success and failure factors ( Adhiarna et al. , 2013 ; Park et al. , 2017 ), and to select technology providers ( Chang et al. , 2012 ).

4.1 Analytic hierarchy process: a brief overview

AHP is a multi-criteria decision-making approach that systematically analyses the complex situation and organises into components of a hierarchical structure ( Saaty, 1990 ). In this study, AHP analysis is conducted in three stages. The first stage involves the identification of critical challenge categories and the challenges of the implementation of blockchain technology and structuring them in hierarchical levels. In the second stage, data is collected through pairwise comparison of the challenges in terms of their importance to a challenge category in the next higher level. Through a comparison of challenges, several preference (square) matrices are generated. For a set of n challenges in a matrix, ( n 2  −  n )/2 judgements are needed, and the remaining judgements are reciprocals ( a ji  = 1/a ij ). We explain the data collection procedure in detail in Section 4.3 . Finally, in stage 3 unique and normalised vectors of the criticality of challenges are computed. The overall criticality of the challenges is determined by aggregating the weights throughout the hierarchy.

Once the criticality is determined, it is important to check the consistency of judgements elicited from the managers. Consistency ratio (CR) is used to measure the extent to which an established preference is retained. A CR ≤ 0.1 is recommended as acceptable ( Saaty, 1990 ). If CR > 0.1, it is suggested that the managers need to re-evaluate their judgements. Overall, the use of AHP to analyse the data satisfies the credibility and dependability criteria of the qualitative research proposed by Shah and Corley (2006) (refer to Table 3 ). In this study, Expert Choice® software is used to calculate the priority weights of challenges and challenge categories.

4.2 Case study

We adopt a multi-case approach that employs a combination of methods, including data collection through semi-structured interviews with developers, consultants, and clients and a review of internal and publicly disclosed documents and websites. The use of data from multiple stakeholders yields rich insights into the phenomena under investigation through the comparison of data from different cases ( Shah and Corley, 2006 ).

In this study, to select the cases that have experience in blockchain implementation, we adopt a two-stage approach. First, from the list of the Top IT Consulting Firms for 2018 in India ( Goodfirms, 2018 ), we distributed questionnaires and the participation criteria to the 50 top consultant and software developing organisations out of which 10 expressed their interest. Out of these 10 firms, finally, six firms who have been involved in blockchain implementation participated in the study. Based on the respondents’ role in technology implementation, we clustered the organisations into two groups, one with three developers and the other with three consultants. Second, to identify their clients, we reviewed the publicly disclosed documents of the respondent consultant and developer organisations. Based on this review, we contacted four clients who have implemented blockchain technology in their Indian operations in the supply chain context and two showed their willingness to participate in the study. Finally, we identified a total of eight respondents, namely, three developers, three consultants, and two clients, as the respondents for this study. As all these cases are involved in blockchain implementation and provide in-depth insights to address the study objectives, the number of cases does not have any impact on the study findings ( Gammelgaard, 2017 ; Mirkovski et al. , 2019 ). These selected cases clustered into different stakeholder groups assist in conducting cross-case analysis that provides crucial information on collaboration among these firms to promote blockchain implementation.

A sample size of eight respondents is deemed to be adequate, given that studies that use the AHP technique are usually conducted with fewer responses from senior executives who are knowledgeable on the issue under investigation. For instance, Abdulrahman et al. (2014) interviewed five experts to ascertain priorities regarding the reverse logistics factors, Sangka et al. (2019) interviewed ten respondents to identify the managerial competencies of third-party logistics providers, and Rahman et al. (2019) used data from five respondents to examine the challenges faced by multinational third-party logistics providers. In our study, we selected respondents who are familiar with the functionalities of blockchain technology and are experienced in the implementation of this technology as well as other technologies, such as IoT, RFID, and ERP. The selection of leading firms in the field with experience in the implementation of blockchain technology as a case study organisations and researchers’ experiences in technology implementations in supply chain assists in meeting the credibility criterion of the qualitative research (see Table 3 ). For confidentiality reasons, all firms are identified using pseudonyms.

4.2.1 Case stakeholder - developers

To maintain the anonymity of the respondents, the three developer companies included in this study are identified as Developer A1, Developer A2 and Developer A3. Developer A1 is India’s oldest (established in the year 1968) and largest IT service, consulting, and business solutions company that has more than 400,000 employees globally. Since it is the largest and oldest technology provider, Developer A1 has the potential to attract most of its existing clients to its blockchain platforms. Developer A2 is a US-based organisation and more than 70% of its 350,000 employees are located outside that country, including 130,000 employees in India. It is one of the first few organisations worldwide to have developed blockchain solutions and is known for providing the infrastructure required for blockchain technology. Developer A3 is a relatively new firm established in 2000 and offers blockchain solutions to large multinational corporations headquartered in India. All the respondents of the developer companies are in a senior executive position in the firm with an average of 15 years of experience and are involved in blockchain implementation at client’s facilities for more than 3 years (see Table 4 ).

4.2.2 Case stakeholder: consultancy firms

The three consultancy firms considered in this study are identified as Consultant B1, Consultant B2 and Consultant B3 to maintain respondent anonymity. Consultant B1, established in 1845, is the oldest financial and advisory service consultancy company, and it started offering blockchain consulting services to financial institutions in 2016. In the study context, Consultant B1 offers blockchain services to Client C2 and their trading partners, whereas Consultant B2 and Consultant B3 started offering their blockchain services to several industries in 2017. Unlike Consultant B1 and Consultant B2 who mostly offer only consultancy services, Consultant B3 also offers software solutions and is currently recognised as India’s second-largest IT provider. All the respondents have experience of over 13 years in offering consultancy services to businesses across multiple industries ranging from financial to IT services (see Table 4 ).

4.2.3 Case stakeholder: client organisations

For confidentiality reasons, the client firms are identified as Client C1 and Client C2. Client C1, established in 1907, is Asia’s first private steel company with fully integrated operations from mining to the manufacturing and marketing of finished products. Client C1 has implemented blockchain technology in collaboration with SAP, and Developer A2 to trace the life cycle of steel bars in its supply chain. Client C2 is a Fortune 500 company and India’s largest bank with operations in more than 36 countries. It is among the initial founding members of the “BankChain platform” for implementing blockchain solutions in Indian banks. Currently, Client C2 uses blockchain for recording and sharing customer information, authenticating contracts, making cross-border payments, and financing trading/supply chain organisations. Respondents from the client organisations are senior executives with over 16 years of experience and have worked on blockchain implementation project for over two years. Over the years, respondents are involved in projects implementing RFID, ERP, and IoT systems in organisations and their supply chains.

In the study context, there are several instances when Consultant B2 and Developer A2 have formed a consortium to offer blockchain services to organisations across several industries. Consultant B1 analysed Client C2’s business goals and identified the applicability of blockchain technology to this client’s existing business ecosystem. These interrelationships among the case study organisations facilitate the collection of insights into not only each technology implementation but also their interactions.

4.3 Data collection

During the interview, respondents were briefed about the study context of blockchain technology implementation in supply chains. Consultants and developers were asked to address the interview questions with the blockchain implementation at client’s supply chain in mind; whereas, respondents from client organisations answered the questions in relation to the blockchain implementation in their supply chains. A three-part questionnaire is used to conduct semi-structured interview. Part A contains questions (in the AHP format) designed to capture the respondents’ opinions on the pairwise comparison of criticality of the challenges and challenge categories. A 9-point rating scales linguistically described as equally, slightly, moderately, strongly and extremely critical corresponding to the values of 1, 3, 5, 7 and 9, respectively is used to capture the degree of criticality of a challenge or challenge-category. Since the respondents were not familiar with the AHP data collection procedure, we provided them a clear explanation, through an example, about the scale and the assignment of criticality scores while making pairwise comparisons between any two challenges. Questions in Part B captures the respondents’ assessments of the effort required to manage the blockchain implementation challenges at clients’ facilities through a scale that ranges from 1 for “least effort required” to 9 for “most effort required”. In addition, respondents were asked to provide justifications for the ratings given while answering Part A and Part B of the questionnaire. Appendices 1 and 2 provides direct quotations of the relevant justifications given by the respondents during the interview. Lastly, Part C contains general questions about the company and the respondent’s background. The duration of the interviews ranged from 90 to 120 min with a short break in between. All the interview transcripts were sent to the participants for feedback, and follow-up conversations with them assisted in providing credibility to the qualitative study. Overall, the set of specific actions taken into consideration while designing this study assists in meeting the credibility, transferability, dependability and conformability criteria that bring rigour to qualitative research (see Table 3 ).

5. Results and analysis

The results are summarised in Tables 5 and 6 . The CR values presented in these tables of each respondent category are within the acceptable limit (i.e. ≤0.1), thus demonstrating that the respondents’ opinions are consistent.

5.1 Identification of critical challenges

The analysis results show that all the developers indicate that organisation (weight = 0.455) is the most critical challenge category with technical expertise (weight = 0.370) as a critical challenge that needs to be addressed. The technology (weight = 0.404) challenge category is next, and in it, complexity (weight = 0.376) is a critical challenge. The less critical challenge category, external environment (weight = 0.073), has security (weight = 0.659) as a critical challenge. Partner’s power (weight = 0.471) is a critical challenge under the least critical challenge category, interorganisational relationships (weight = 0.065).

All the consultants indicate that the technology challenge category (weight = 0.574) is more critical than the organisational (weight = 0.254), external environment (weight = 0.115), and interorganisational relationships (weight = 0.056) challenge categories. According to them, the most important challenge in each challenge category is as follows: complexity (weight = 0.382) under the technology challenge category; technical expertise (weight = 0.426) in the organisational challenge category; security (weight = 0.584) in the external environment challenge category; and partner’s power (weight = 0.365) under the interorganisational relationship challenge category (see Table 5 ).

By contrast, all clients indicate that technology is the most critical challenge category (weight = 0.518) with complexity (weight = 0.391) as the most critical challenge. It is followed by the organisational challenge category (weight = 0.330) with top-management support (weight = 0.326) as the most critical challenge; the external environment category (weight = 0.103) with security (weight = 0.455) and government regulation (weight = 0.455) as the most critical challenges; and the interorganisational relationship challenge category (weight = 0.048) with partner’s power (weight = 0.474) as the most critical challenge.

Our clients express that it is not easy for their organisation to consider the blockchain technology because of the initial costs associated with the technology and lack of awareness on operational costs of running the technology.

When our clients adopt blockchain technology in full-scale to trace the products it is important to integrate with the ERP systems of all the supply chain members.

In most cases, the opinions of the respondents are consistent with the overall judgement. Some exceptions are observed: for instance, Consultant B1 emphasises top-management support (weight = 0.368) and financial resources (weight = 0.130) from the organisational challenge category, whereas Consultant B2 and Consultant B3 prioritise compatibility and complexity from the technology challenge category as critical challenges (see Table 6 ).

5.2 Level of effort required to overcome the challenges

Compatibility of blockchain with the existing systems and IOT devices for data transfer is possible through the applications such as enterprise application adapters.

The results obtained on analysing all the consultants’ judgements indicate that they consider that challenges related to complexity (weight = 0.114); financial resources (weight = 0.100); and compatibility, observability, and government regulations which have equal weight (0.087) require increased effort. However, they indicate that privacy (weight = 0.013); trust (weight = 0.039); and security, partner’s power and technical expertise (all with weight = 0.044) are the challenges that require less effort. At the individual level, there are some differences in opinions, such as Consultant B2’s (weight = 0.069) perception that the effort required to address the challenge related to industry characteristics is high compared with the perceptions of Consultant B1 (weight = 0.057) and Consultant B3 (weight = 0.056).

The results obtained on analysing all the clients’ judgements indicate that privacy (weight = 0.095), technical expertise (weight = 0.089) and complexity (weight = 0.089) are the challenges they view as requiring more effort to address. Conversely, they consider that government regulations (weight = 0.039), industry characteristics (weight = 0.034) and partner’s power (weight = 0.034) are the challenges that require less effort. Among the clients, there are some similarities and differences in opinion. One such difference is that Client C2 (weight = 0.087) considers that a strong effort is required to address the firm size challenge, whereas Client C1 (weight = 0.057) views it as requiring relatively less effort.

5.3 Classification of challenges based on the criticality–effort matrix

Next, we perform a criticality–effort matrix analysis using a 2 × 2 format. “Implement immediately”, “plan to execute”, “seek assistance”, and “no action required for now” are the four quadrants of the matrix. The challenges in the “implement immediately” quadrant are critical for implementing blockchain technology and require less effort. The “plan to execute” quadrant challenges are critical but require great effort to address. By contrast, the challenges in the “seek assistance” quadrant are not critical and require less effort; therefore, these challenges can be addressed by seeking external assistance or outsourcing. However, the challenges in the “no action required for now” quadrant are less critical to blockchain implementation and require more effort to address than other challenges. Hence, these challenges should be addressed last, that is, no action is required at the initial stage. In this study, based on the perceptions of the developer, consultant, and client groups we draw three matrices (see Figure 2 ).

According to the developers’ perspective, the challenges of technical expertise, firm size, and security belong to the “implement immediately” quadrant. Top-management support, financial resources, complexity, compatibility, and partner’s power are the challenges in the “plan to execute” quadrant. The consultants perceive technical expertise and security as the critical challenges that require less effort (these belong to the “implement immediately” quadrant). The challenges related to complexity, compatibility, relative advantages, observability, and top-management support are grouped under the quadrant “plan to execute”. From the clients’ perspective, the challenges of trialability, relative advantages, and observability belong to the “implement immediately” quadrant. Top-management support, technical expertise, financial resources, complexity, and compatibility are the challenges in the quadrant “plan to execute” (see Figure 2 ).

6. Discussion and implications

6.1 discussion.

The analysis results indicate similarities and differences in the perceptions of stakeholder groups regarding the criticality of the challenges and the effort required to address these challenges. These differences in perceptions result in variations in the criticality–effort matrix and highlight the need for adopting different strategies to ensure successful technology implementation.

6.1.1 The “implement immediately” quadrant

Technically skilled professionals are critical. Salaries and demand for blockchain technology professionals are very high. So, it is difficult to attract and retain people with blockchain technology skills.

It is not easy to convince current employees to undergo training to use blockchain technology in their processes.

However, all the respondents believe it would be much easier to find skilled employees in the near future because educational institutes in India, such as the Indian Institute of Technology, have started offering blockchain-based courses.

We are offering services to audit the mining practice and protect the confidentiality, integrity, and availability of the system and its information to promote our client’s confidence.

Hyperledger Fabric and smart contracts that we developed can be integrated with the IoT/sensors and smart tags to capture and record data automatically thus eliminating human errors.

As the consultants and developers have started offering new services to address the security issues associated with blockchain technology, it is perceived that the effort required by client to address the blockchain implementation challenge is less. In comparison to consultancies and developers, the client organisations placed the challenges of trialability, relative advantages and observability of the technology challenge category under the “implement immediately” quadrant.

6.1.2 The “plan to execute” quadrant

The lack of a knowledgeable and skilled workforce to implement the technology has demotivated us from its implementation.

Moreover, insufficient understanding of how technology fits with the organisation’s policy and benefits offered by the technology results in a low-level of support. Especially, as this technology is still evolving and changing continually, top management needs to offer different forms of support for different aspects. For example, Developer A2 emphasised the need for the Client’s top management to support employees undergoing the change management process. Given that 25% of organisations worldwide are replacing existing legacy systems with blockchain solutions ( Deloitte, 2019 ), top-management support is critical.

Our supply chains are complex and constantly expanding requiring multiple systems from various organisations to be compatible.

The structure of our client is so complex, with duplications in roles across several departments, it made it challenging for us to implement blockchain technology.

Meanwhile, Consultant B2 highlights that its complexity makes it difficult for its end users who are external to the organisation to appreciate the benefits that this technology offers, which affects its uptake in supply chain. To remove the complexity in implementing and running blockchain networks at their clients, Developer A1 and Developer A3 are working on creating templates, or in other words, blockchain-as-a-service. Although the development of these services requires more resources and careful planning, these will motivate more organisations to consider using blockchain technology.

6.1.3 The “seek assistance” quadrant

This technology is a democratisation of trust. However, it is abstract, understanding it requires technical knowledge. Many of the related processes are not transparent, which makes it difficult for our clients to realise its benefits and thus results in trust issues.

Therefore, consultants and developers are developing ecosystems to promote trust regarding technology implementation among their clients. Meanwhile, client organisations should seek consensus among the supply chain members to build trust.

Successful implementation of blockchain technology in supply chains depends upon the influencing capability of the dominating player.

Over the past year, Consultant B2 has worked closely with large retailers who have influenced all their supply chain members towards blockchain implementation so that they can trace the origin of the products they are selling. Meanwhile, Client C1 requires all its supply chain members to use blockchain technology to trace the life cycle of steel bars.

Blockchain implementation in India is not influenced by government regulations. But, the recognition of blockchain by the Indian government will provide confidence to organisations that intend to adopt the technology.

This is evident in the case of the blockchain implementation by Client C2 who has persuaded private banks to consider blockchain technology. Moreover, government support initiatives, such as providing training and incentives and facilitating research and development have enabled the adoption of technologies such as RFID in supply chains ( Cole et al. , 2019 ). In the study context, the Indian government’s plan of preparing a national blockchain framework will facilitate the wider deployment of this technology ( NITI Aayog, 2020 ).

The successful deployment of blockchain technology in supply chains depends on the relative advantages it offers, that is, organisations’ ability to perceive the greater benefits of this new technology compared with previous technologies ( Lielacher, 2018 ). In the study context, the participating consultants and developers believe that the lack of evidence on relative advantages is not a critical challenge that influences the technology implementation and it requires the active disclosure by firms that have successfully implemented the technology. Consultant B1 and Consultant B2 have highlighted in their interviews that they are successful in offering blockchain solutions across several industries and have published white papers illustrating the increased benefits offered by the technology compared with earlier technologies. In addition, client organisations believe that they must understand the relative advantages offered by the technology, and hence, they rely on their consultants’ white papers at this stage.

6.1.4 The “no action required for now” quadrant

There are no challenges common to all stakeholders in the quadrant “no action required for now”. Among all these challenges, the consultants and the developers perceive that the characteristics of industry affect blockchain implementation. Since its introduction, blockchain technology has been implemented in the financial industry. In 2018, the financial industry accounted for 45% of the global blockchain expenditure ( IDC, 2019 ). More recently, it has been implemented in the technology, media, telecommunication, and non-food manufacturing sectors ( Deloitte, 2019 ). Further expansion to other industries requires efforts from all the stakeholder groups on educating managers about the benefits the technology offers, and this approach will assist in meeting the projected blockchain expenditure of US$ 12.4 billion by 2023 ( IDC, 2019 ). Moreover, most of the interview respondents rated the privacy challenge as less critical because they believe the development of permissioned blockchain with limited data access to participants will address this challenge. However, significant effort and careful planning are required to customise blockchain to assign rights to respondents to review only permissible parts.

6.2 Implications

6.2.1 practical implications.

By performing criticality–effort matrix analysis, this study offers several managerial implications. First, it provides strategies to developers for bringing about advancements in blockchain technology that would improve supply chain efficiency when implemented. This study identifies the lack of technical expertise to promote technology development as a major challenge concerning developers. Therefore, developers need to focus on retaining skilled people through employee recognition programs. However, it is a major challenge in the Indian context because the IT industry accounts for a significant extent of the brain drain issue that the country experiences.

Second, this study’s findings would assist consultants in developing plans to facilitate the implementation of blockchain technology in their clients’ supply chains. The consultants identified the challenges related to the workforce as employees who either have no understanding of the technology or have no cross-industry work experience. Consultants can only acquire cross-industry skills over time with experience. Thus, consultancies need to develop an internal digital skills program to boost the technological skills of their employees. The other challenge that needs to be immediately addressed is security issues related to data mining in the blockchain. This challenge provides consultants with an opportunity to develop services to audit the mining practice and protect the confidentiality, integrity, and availability of the system and its information.

Third, the findings of this study would facilitate organisations to develop ways for the implementation of blockchain technology in supply chains. The study results indicate that trialability, observability, and the relative advantages of the technology facilitate organisations to opt for technology implementation in supply chains. For successful blockchain implementation, organisations should form consortia with the other firms offering similar business services. Joining consortia is critical because it yields cost savings and accelerates learning among the stakeholders ( Deloitte, 2019 ).

Lastly, the study findings offer policy implications. This study identifies that the lack of regulation does not influence technology uptake. However, when government organisations implement technology it motivates the others in the industry towards technology implementation. Therefore, government organisations should take the lead in technology implementation in supply chains. In the context of developed nations, regulation plays an important role in their technology uptake, which can be observed in the case of earlier technologies, such as RFID. Hence, developing a regulatory framework would assist in the technology uptake in developed economies. Although blockchain implementation in India is not governed by any regulations, the Indian government has proposed a skill development initiative to supply the much-needed skilled workforce trained on blockchain technology. The availability of such a skilled workforce would enable India to become the next Silicon Valley.

6.2.2 Theoretical implications

The study contributes to existing research in three ways. Blockchain technology offers several benefits to emerging economies such as India where transparency is an issue ( Queiroz and Wamba, 2019 ). Despite the benefits offered by the technology, its adoption depends on how the implementation challenges are addressed ( Kouhizadeh et al. , 2021 ). The current literature on blockchain technology in supply chains has highlighted the need for research that investigates the blockchain implementation challenges through examining the real cases who have implemented the technology ( Caldarelli et al. , 2021 ; Kshetri, 2021 ). This is the first to identify and prioritise the challenges from the perspectives of multiple stakeholder groups involved in blockchain technology implementation. The use of insights from diverse stakeholder groups provides crucial insights into a well-researched concept such as the challenges of technology implementation ( Gammelgaard, 2017 ). By investigating the challenges that developers, consultants, and client organisations need to address, this study provides a holistic understanding and facilitates coordination among these stakeholder groups for successful blockchain implementation.

The second implication of the study is to theory. The selection of the TOE theory as an underpinning theory addresses the need identified by Hald and Kinra (2019) for research to adapt to organisational theory to explore the implementation of blockchain technology in supply chains. Unlike the implementation of technology in an organisation which is influenced by technology characteristics and organisational context, technology implementation in the supply chain is complex where the focal firm and its relationship with other firms determine the implementation ( Kouhizadeh et al. , 2021 ). To examine the impact of relationships among supply chain members on the implementation of blockchain technology, this study extends the TOE framework by incorporating the interorganisational relationships challenge category to examine supply chain relationships. The use of interorganisational relationships challenge category to extend theory classifies this study as the theory elaboration approach of case study analysis ( Ketokivi and Choi, 2014 ).

The third implication of the study is examining the blockchain implementation challenges in developing countries. In the literature industry characteristics, trust, information sharing, and privacy among the supply chain members are identified as the critical challenges that needs to be addressed for the successful implementation of blockchain technology in supply chains ( Kshetri, 2021 ; Wang et al. , 2019b ; Queiroz and Wamba, 2019 ; Kouhizadeh et al. , 2021 ). Whereas, our study findings indicate that these challenges are less critical in the context Indian supply chains. Rather findings of our study highlight complexity, compatibility, top management support, and technical expertise as the critical challenges. Among these critical challenges consultants and developers believe technical expertise is considered as the challenge that requires less effort to be addressed. Meanwhile, in India motivating top management to adopt new technologies that has long-term benefits and minimising the complexity associated with the informal structure in organisations requires significant effort. Thus, from theoretical perspective researchers need to focus on the challenges related to technological and organisational categories while investigating the blockchain implementation in the context of developing countries rather than the challenges related to external environment and interorganisational relationships.

7. Limitations and future research

Despite the significance of its results, this study has some limitations. The major limitation is the study context. The results were obtained by focusing on the Indian context and therefore may not be applicable to other developing nation contexts. In the future, more research is needed in other developing countries. In addition, developed countries, such as the United States and Western European nations, are expected to account for 39.7% and 24.4% respectively of the global blockchain expenditure by 2023 ( Leader insights, 2019 ), and thus, researchers need to empirically investigate the technology implementation challenges in the developed nation context.

Given the newness, very few firms in India have implemented blockchain technology. Thus, in this study the sample size of the respondents with experience in implementing blockchain technology is limited. Broader implementation of technology in the future will assist to conduct study with a larger client sample and their supply chain members. The other limitation related to the study sample is respondent groups. This study offered a comparative analysis of the criticality of blockchain implementation challenges from the perspectives of developers, consultants, and client organisations that are a part of different consortia involved in blockchain implementation. However, such consortia are increasingly being formed by firms offering similar solutions to gain cost advantages and accelerate blockchain learning. Therefore, to provide a holistic understanding of blockchain implementation, future research needs to consider the views of all the consortia members, including those of competitors or other industry members, to identify other critical challenges.

Conceptual model of challenges of blockchain implementation in supply chains

Criticality-effort matrix for the implementation of blockchain technology at the client’s supply chains

Role of blockchain technology in supply chain function

Literature used TOE framework to examine technology adoption in supply chains

Trustworthiness criteria of research

Profile of case study companies and respondents

Relative weights of challenge-categories and challenges of consultants, developers, and clients

Overall priority weights of blockchain implementation challenges at client’s facilities perceived by consultants, developers, and clients

Effort required by developers, consultants, and clients to address the blockchain implementation challenges at client’s facilities

Appendix 1 Examples of interview evidence on rating the challenge criticality

Appendix 2 examples of interview evidence on rating the effort required.

Abdulrahman , M.D. , Gunasekaran , A. and Subramanian , N. ( 2014 ), “ Critical barriers in implementing reverse logistics in the Chinese manufacturing sectors ”, International Journal of Production Economics , Vol.  147 , pp.  460 - 471 .

Adhiarna , N. , Hwang , Y.M. , Park , M.J. and Rho , J.J. ( 2013 ), “ An integrated framework for RFID adoption and diffusion with a stage-scale-scope cubicle model: a case of Indonesia ”, International Journal of Information Management , Vol.  33 No.  2 , pp.  378 - 389 .

Babich , V. and Hilary , G. ( 2020 ), “ OM Forum- Distributed ledgers and operations: what operations management researchers should know about block chain technology ”, Manufacturing and Service Operations Management , Vol.  22 No.  2 , pp.  223 - 240 .

Baker , J. ( 2011 ), “ The technology–organization–environment framework ”, in Dwivedi , Y. , Wade , M. and Schneberger , S. (Eds), Information Systems Theory: Explaining and Predicting Our Digital Society , Springer , New York , Vol.  1 , pp. 231 - 246 .

Bigdeli , A.Z. , Kamal , M.M. and De Cesare , S. ( 2013 ), “ Electronic information sharing in local government authorities: factors influencing the decision-making process ”, International Journal of Information Management , Vol.  33 No.  5 , pp.  816 - 830 .

Bradford , M. , Earp , J.B. and Grabski , S. ( 2014 ), “ Centralized end-to-end identity and access management and ERP systems: a multi-case analysis using the Technology Organization Environment framework ”, International Journal of Accounting Information Systems , Vol.  15 No.  2 , pp.  149 - 165 .

Caldarelli , G. , Zardini , A. and Rossignoli , C. ( 2021 ), “ Blockchain adoption in the fashion sustainable supply chain: pragmatically addressing barriers ”, Journal of Organizational Change Management , Vol.  34 No.  2 , pp.  507 - 524 .

Casey , M.J. and Wong , P. ( 2017 ), “ Global supply chains are about to get better, thanks to blockchain ”, Harvard Business Review , Vol.  13 , pp.  1 - 6 .

Chan , F.T. , Chong , A.Y.L. and Zhou , L. ( 2012 ), “ An empirical investigation of factors affecting e-collaboration diffusion in SMEs ”, International Journal of Production Economics , Vol.  138 No.  2 , pp.  329 - 344 .

Chang , S.I. , Yen , D.C. , Ng , C.S.P. and Chang , W.T. ( 2012 ), “ An analysis of IT/IS outsourcing provider selection for small-and medium-sized enterprises in Taiwan ”, Information and Management , Vol.  49 No.  5 , pp.  199 - 209 .

Chong , A.Y.L. and Ooi , K.B. ( 2008 ), “ Adoption of interorganizational system standards in supply chains: an empirical analysis of RosettaNet standards ”, Industrial Management and Data Systems , Vol.  108 No.  4 , pp.  529 - 547 .

Cole , R. , Stevenson , M. and Aitken , J. ( 2019 ), “ Blockchain technology: implications for operations and supply chain management ”, Supply Chain Management: An International Journal , Vol.  24 No.  4 , pp.  469 - 483 .

Deloitte ( 2019 ), “ Deloitte’s 2019 global blockchain survey ”, available at: https://www2.deloitte.com/content/dam/Deloitte/se/Documents/risk/DI_2019-global-blockchain-survey.pdf ( accessed 13 March 2020 ).

DHL ( 2019 ), “ Blockchian: is blockchain the new internet? ”, available at: https://www.logistics.dhl/in-en/home/insights-and-innovation/insights/blockchain.html ( accessed 23 March 2020 ).

Gammelgaard , B. ( 2017 ), “ The qualitative case study ”, The International Journal of Logistics Management , Vol.  28 No.  4 , pp.  910 - 913 .

Gaur , N. ( 2020 ), “ Blockchain challenges in adoption ”, Managerial Finance , Vol.  46 No.  6 , pp.  849 - 858 .

Goodfirms ( 2018 ), “ Top IT consulting companies ”, available at: https://www.goodfirms.co/it-services/it-consulting?sort_by=any&location[51]=in ( accessed 15 February 2018 ).

Gouda , S.K. and Saranga , H. ( 2018 ), “ Sustainable supply chains for supply chain sustainability: impact of sustainability efforts on supply chain risk ”, International Journal of Production Research , Vol.  56 No.  17 , pp.  5820 - 5835 .

Gurtu , A. and Johny , J. ( 2019 ), “ Potential of blockchain technology in supply chain management: a literature review ”, International Journal of Physical Distribution and Logistics Management , Vol.  49 No.  4 , pp.  881 - 900 .

Hald , K.S. and Kinra , A. ( 2019 ), “ How the blockchain enables and constrains supply chain performance ”, International Journal of Physical Distribution and Logistics Management , Vol.  49 No.  4 , pp.  376 - 397 .

Huang , Z. , Janz , B.D. and Frolick , M.N. ( 2008 ), “ A comprehensive examination of Internet-EDI adoption ”, Information Systems Management , Vol.  25 No.  3 , pp.  273 - 286 .

IDC ( 2019 ), “ New IDC spending guide sees strong growth in blockchain solutions leading to $15.9 billion market in 2023 ”, available at: https://www.businesswire.com/news/home/20190808005064/en/New-IDC-Spending-Guide-Sees-Strong-Growth-in-Blockchain-Solutions-Leading-to-15.9-Billion-Market-in-2023 (accessed 3 October 2022) .

Kamble , S. , Gunasekaran , A. and Arha , H. ( 2019 ), “ Understanding the Blockchain technology adoption in supply chains-Indian context ”, International Journal of Production Research , Vol.  57 No.  7 , pp.  2009 - 2033 .

Kaplan , B. and Duchon , D. ( 1988 ), “ Combining qualitative and quantitative methods in information systems research: a case study ”, MIS Quarterly , Vol.  12 No.  4 , pp. 571 - 586 .

Ketokivi , M. and Choi , T. ( 2014 ), “ Renaissance of case research as a scientific method ”, Journal of Operations Management , Vol.  32 No.  5 , pp. 232 - 240 .

Kouhizadeh , M. , Saberi , S. and Sarkis , J. ( 2021 ), “ Blockchain technology and the sustainable supply chain: theoretically exploring adoption barriers ”, International Journal of Production Economics , Vol.  231 , 107831 .

Kshetri , N. ( 2021 ), “ Blockchain and sustainable supply chain management in developing countries ”, International Journal of Information Management , Vol.  60 , p. 102376 .

Leader insights ( 2019 ), “ IDC sees trade finance as biggest blockchain application in China ”, available at: https://www.ledgerinsights.com/idc-trade-finance-biggest-enterprise-blockchain-app-china/ ( accessed 10 February 2020 ).

Lielacher ( 2018 ), “ Five challenges blockchain technology must overcome before mainstream adoption ”, available at: https://www.nasdaq.com/articles/five-challenges-blockchain-technology-must-overcome-mainstream-adoption-2018-01-03 ( accessed 10 April 2019 ).

Lin , H.F. ( 2014 ), “ Understanding the determinants of electronic supply chain management system adoption: using the technology–organization–environment framework ”, Technological Forecasting and Social Change , Vol.  86 , pp.  80 - 92 .

Low , C. , Chen , Y. and Wu , M. ( 2011 ), “ Understanding the determinants of cloud computing adoption ”, Industrial Management and Data Systems , Vol.  111 No.  7 , pp.  1006 - 1023 .

Mirkovski , K. , Davison , R.M. and Martinsons , M.G. ( 2019 ), “ The effects of trust and distrust on ICT-enabled information sharing in supply chains: evidence from small-and medium-sized enterprises in two developing economies ”, The International Journal of Logistics Management , Vol.  30 No.  3 , pp. 892 - 926 .

Nandi , M.L. , Nandi , S. , Moya , H. and Kaynak , H. ( 2020 ), “ Blockchain technology-enabled supply chain systems and supply chain performance: a resource-based view ”, Supply Chain Management: An International Journal , Vol.  25 No.  6 , pp.  841 - 862 .

NITI Aayog ( 2020 ), “ Blockchain: the India strategy ”, available at: https://niti.gov.in/sites/default/files/2020-01/Blockchain_The_India_Strategy_Part_I.pdf ( accessed 5 September 2020 ).

52 Park , J.H. , Kim , Y.B. and Kim , M.K. ( 2017 ), “ Investigating factors influencing the market success or failure of IT services in Korea ”, International Journal of Information Management , Vol.  37 No.  1 , pp.  1418 - 1427 .

Perdana , A. , Robb , A. , Balachandran , V. and Rohde , F. ( 2021 ), “ Distributed ledger technology: its evolutionary path and the road ahead ”, Information and Management , Vol.  58 No.  3 , 103316 .

Pournader , M. , Shi , Y. , Seuring , S. and Koh , S.L. ( 2020 ), “ Blockchain applications in supply chains, transport and logistics: a systematic review of the literature ”, International Journal of Production Research , Vol.  59 No.  7 , pp.  2063 - 2081 .

Queiroz , M.M. and Wamba , S.F. ( 2019 ), “ Blockchain adoption challenges in supply chain: an empirical investigation of the main drivers in India and the USA ”, International Journal of Information Management , Vol.  46 , pp.  70 - 82 .

Queiroz , M.M. , Wamba , S.F. , De Bourmont , M. and Telles , R. ( 2021 ), “ Blockchain adoption in operations and supply chain management: empirical evidence from an emerging economy ”, International Journal of Production Research , Vol.  59 No.  20 , pp.  6087 - 6103 .

Rahman , S. , Ahsan , K. , Yang , L. and Odgers , J. ( 2019 ), “ An investigation into critical challenges for multinational third-party logistics providers operating in China ”, Journal of Business Research , Vol.  103 , pp.  607 - 619 .

Rogers , E.M. ( 1995 ), Diffusion of Innovations , 4th ed. , The Free Press , New York .

Saaty , T.L. ( 1990 ), “ An exposition of the AHP in reply to the paper Remarks on the analytic hierarchy process ”, Management Science , Vol.  36 No.  3 , pp.  259 - 268 .

Saberi , S. , Kouhizadeh , M. , Sarkis , J. and Shen , L. ( 2019 ), “ Blockchain technology and its relationships to sustainable supply chain management ”, International Journal of Production Research , Vol.  57 No.  7 , pp.  2117 - 2135 .

Sangka , B.K. , Rahman , S. , Yadlapalli , A. and Jie , F. ( 2019 ), “ Managerial competencies of 3PL providers: a comparative analysis of Indonesian firms and multinational companies ”, The International Journal of Logistics Management , Vol.  30 No.  4 , pp.  1054 - 1077 .

Shah , S.K. and Corley , K.G. ( 2006 ), “ Building better theory by bridging the quantitative–qualitative divide ”, Journal of Management Studies , Vol.  43 No.  8 , pp.  1821 - 1835 .

Tokkozhina , U. , Lucia Martins , A. and Ferreira , J.C. ( 2022 ), “ Uncovering dimensions of the impact of blockchain technology in supply chain management ”, Operations Management Research , pp.  1 - 27 .

Tornatzky , L.G. and Fleischer , M. ( 1990 ), The Processes of Technological Innovation , Lexington Books , Lexington, MA .

Upadhyay , N. ( 2020 ), “ Demystifying blockchain: a critical analysis of challenges, applications and opportunities ”, International Journal of Information Management , Vol.  54 , 102120 .

Vivaldini , M. ( 2021 ), “ Blockchain in operations for food service distribution: steps before implementation ”, International Journal of Logistics Management , Vol.  32 No.  3 , pp.  995 - 1029 .

Wang , Y. , Han , J.H. and Beynon-Davies , P. ( 2019a ), “ Understanding blockchain technology for future supply chains: a systematic literature review and research agenda ”, Supply Chain Management: An International Journal , Vol.  24 No.  1 , pp.  62 - 84 .

Wang , Y. , Singgih , M. , Wang , J. and Rit , M. ( 2019b ), “ Making sense of blockchain technology: how will it transform supply chains? ”, International Journal of Production Economics , Vol.  211 , pp.  221 - 236 .

Westbrook , A. and Braver , T.S. ( 2015 ), “ Cognitive effort: a neuroeconomic approach ”, Cognitive, Affective, and Behavioral Neuroscience , Vol.  15 No.  2 , pp.  395 - 415 .

Wong , L.-W. , Tan , G.W.-H. , Lee , V.-H. , Ooi , K.-B. and Sohal , A. ( 2020a ), “ Unearthing the determinants of blockchain adoption in supply chain management ”, International Journal of Production Research , Vol.  58 No.  7 , pp.  2100 - 2123 .

Wong , L.W. , Leong , L.Y. , Hew , J.J. , Tan , G.W.H. and Ooi , K.B. ( 2020b ), “ Time to seize the digital evolution: adoption of blockchain in operations and supply chain management among Malaysian SMEs ”, International Journal of Information Management , Vol. 52 , 101997 .

Yin , R.K. ( 2003 ), “ Designing case studies ”, Qualitative Research Methods , Vol.  5 No.  14 , pp. 359 - 386 .

Corresponding author

Related articles, we’re listening — tell us what you think, something didn’t work….

Report bugs here

All feedback is valuable

Please share your general feedback

Join us on our journey

Platform update page.

Visit emeraldpublishing.com/platformupdate to discover the latest news and updates

Questions & More Information

Answers to the most commonly asked questions here

• Open access
• Published: 24 May 2024

Integration of case-based learning and three-dimensional printing for tetralogy of fallot instruction in clinical medical undergraduates: a randomized controlled trial

• Jian Zhao 1   na1 ,
• Xin Gong 1   na1 ,
• Jian Ding 1 ,
• Kepin Xiong 2 ,
• Kangle Zhuang 3 ,
• Rui Huang 1 ,
• Shu Li 4 &
• Huachun Miao 1  

BMC Medical Education volume  24 , Article number:  571 ( 2024 ) Cite this article

Metrics details

Case-based learning (CBL) methods have gained prominence in medical education, proving especially effective for preclinical training in undergraduate medical education. Tetralogy of Fallot (TOF) is a congenital heart disease characterized by four malformations, presenting a challenge in medical education due to the complexity of its anatomical pathology. Three-dimensional printing (3DP), generating physical replicas from data, offers a valuable tool for illustrating intricate anatomical structures and spatial relationships in the classroom. This study explores the integration of 3DP with CBL teaching for clinical medical undergraduates.

Sixty senior clinical medical undergraduates were randomly assigned to the CBL group and the CBL-3DP group. Computed tomography imaging data from a typical TOF case were exported, processed, and utilized to create four TOF models with a color 3D printer. The CBL group employed CBL teaching methods, while the CBL-3DP group combined CBL with 3D-printed models. Post-class exams and questionnaires assessed the teaching effectiveness of both groups.

The CBL-3DP group exhibited improved performance in post-class examinations, particularly in pathological anatomy and TOF imaging data analysis ( P  < 0.05). Questionnaire responses from the CBL-3DP group indicated enhanced satisfaction with teaching mode, promotion of diagnostic skills, bolstering of self-assurance in managing TOF cases, and cultivation of critical thinking and clinical reasoning abilities ( P  < 0.05). These findings underscore the potential of 3D printed models to augment the effectiveness of CBL, aiding students in mastering instructional content and bolstering their interest and self-confidence in learning.

The fusion of CBL with 3D printing models is feasible and effective in TOF instruction to clinical medical undergraduates, and worthy of popularization and application in medical education, especially for courses involving intricate anatomical components.

Peer Review reports

Tetralogy of Fallot (TOF) is the most common cyanotic congenital heart disease(CHD) [ 1 ]. Characterized by four structural anomalies: ventricular septal defect (VSD), pulmonary stenosis (PS), right ventricular hypertrophy (RVH), and overriding aorta (OA), TOF is a focal point and challenge in medical education. Understanding anatomical spatial structures is pivotal for learning and mastering TOF [ 2 ]. Given the constraints of course duration, medical school educators aim to provide students with a comprehensive and intuitive understanding of the disease within a limited timeframe [ 3 ].

The case-based learning (CBL) teaching model incorporates a case-based instructional approach that emphasizes typical clinical cases as a guide in student-centered and teacher-facilitated group discussions [ 4 ]. The CBL instructional methods have garnered widespread attention in medical education as they are particularly appropriate for preclinical training in undergraduate medical education [ 5 , 6 ]. The collection of case data, including medical records and examination results, is essential for case construction [ 7 ]. The anatomical and hemodynamic consequences of TOF can be determined using ultrasonography, computed tomography (CT), and magnetic resonance imaging techniques. However, understanding the anatomical structures from imaging data is a slow and challenging psychological reconstruction process for undergraduate medical students [ 8 ]. Three-dimensional (3D) visualization is valuable for depicting anatomical structures [ 9 ]. 3D printing (3DP), which creates physical replicas based on data, facilitates the demonstration of complex anatomical structures and spatial relationships in the classroom [ 10 ].

During the classroom session, 3D-printed models offer a convenient means for hands-on demonstration and communication, similar to facing a patient, enhancing the efficiency and specificity of intra-team communication and discussion [ 11 ]. In this study, we printed TOF models based on case imaging data, integrated them into CBL teaching, and assessed the effectiveness of classroom instruction.

Research participants

The study employed a prospective, randomized controlled design which received approval from the institutional ethics committee. Senior undergraduate students majoring in clinical medicine at Wannan Medical College were recruited for participation based on predefined inclusion criteria. The researchers implemented recruitment according to the recruitment criteria by contacting the class leaders of the target classes they had previously taught. Notably, these students were in their third year of medical education, with anticipation of progressing to clinical courses in the fourth year, encompassing Internal Medicine, Surgery, Obstetrics, Gynecology, and Pediatrics. Inclusion criteria for participants encompassed the following: (1) proficient communication and comprehension abilities, (2) consistent attendance without absenteeism or truancy, (3) absence of failing grades in prior examinations, and (4) capability to conscientiously fulfill assigned learning tasks. Exclusion criteria were (1) absence from lectures, (2) failure to complete pre-and post-tests, and (3) inadequate completion of questionnaires. For their participation in the study, Students were provided access to the e-book “Localized Anatomy,” authored by the investigators, as an incentive for their participation. Voluntary and anonymous participation was emphasized, with participants retaining the right to withdraw from the study at any time without providing a reason.

The study was conducted between May 1st, 2023, and June 30, 2023, from recruitment to completion of data collection. Drawing upon insights gained from a previous analogous investigation which yielded an effect size of 0.95 [ 10 ]. Sample size was computed, guided by a statistical consultant, with the aim of 0.85 power value, predicated on an effect size of 0.8 and a margin of error set at 0.05. A minimum of 30 participants per group was calculated using G*Power software (latest ver. 3.1.9.7; Heinrich-Heine-Universität Düsseldorf, Düsseldorf, Germany), resulting in the recruitment of a total of 60 undergraduate students. Each participant was assigned an identification number, with codes placed in boxes. Codes drawn from the boxes determined allocation to either the CBL group or the CBL-3DP group. Subsequently, participants were randomly assigned to either the CBL group, receiving instruction utilizing the CBL methodology, or the CBL-3DP group, which received instruction integrating both CBL and 3D Printed models.

Printing of TOF models

Figure  1 A shows the printing flowchart of the TOF models. A typical TOF case was collected from the Yijishan Hospital of Wannan Medical College. The CT angiography imaging data of the case was exported. Mimics Research 20.0 software (Mimics Innovation Suite version 20, Materialize, Belgium) was used for data processing. The cardiovascular module of the CT-Heart tool was employed to adjust the threshold range, independently obtain the cardiac chambers and vessels, post-process the chambers and vessels to generate a hollow blood pool, and merge it with the myocardial volume to construct a complete heart model. The file was imported into Magics 24.0 software (version 24.0; Materialize, Belgium) for correction using the Shell tool page. After repairs, the model entered the smoothing page, where tools such as triangular surface simplification, local smoothing, refinement and smoothing, subdivision of components, and mesh painting were utilized to achieve varying degrees of smoothness. Finally, optimized data were obtained and exported as stereolithography (STL) files. An experienced cardiothoracic surgeon validated the anatomical accuracy of the digital model.

The STL files were imported into a 3D printer (J401Pro; Sailner 3D Technology, China) for model printing. This printer can produce full-color medical models using different materials. The models were fabricated using two distinct materials: rigid and flexible. Both materials are suitable for the observational discussion of the teaching objectives outlined in our study. From the perspective of observing pathological changes in the TOF, there is no significant difference between the two materials.

Experimental flow chart of this study. A TOF model printing flow chart. B The instructional framework

Teaching implementation

Figure  1 B illustrates the instructional framework employed in this study. One week preceding the class session, all the students were tasked with a 30-minute self-study session, focusing on the theoretical content related to TOF as outlined in the Pediatrics and Surgery textbooks, along with a review of pertinent academic literature. Both groups received co-supervision from two basic medicine lecturers boasting over a decade of teaching experience, alongside a senior cardiothoracic surgeon. Teaching conditions remained consistent across groups, encompassing uniform assessment criteria and adherence to predefined teaching time frames, all conducted in a Project-Based Learning (PBL) classroom at Wannan Medical College. Additionally, a pre-course examination was administered to gauge students’ preparedness for self-study.

In adherence to the curriculum guidelines, the teaching objectives aimed to empower students to master TOF’s clinical manifestations, diagnostic modalities, and differential diagnoses, while acquainting them with treatment principles and surgical methodologies. Additionally, the objectives sought to cultivate students’ clinical reasoning abilities and problem-solving skills. the duration of instruction for the TOF theory session was standardized to 25 min. The didactic content was integrated with the TOF case study to construct a coherent pedagogical structure.

During the instructional session, both groups underwent teaching utilizing the CBL methodology. Clinical manifestations and case details of TOF cases were presented to stimulate students’ interest and curiosity. Subsequently, the theory of TOF, including its etiology, pathogenesis, pathologic anatomy, clinical manifestations, diagnostic methods, and therapeutic interventions, was briefly elucidated. Emphasis was then placed on the case, wherein selected typical TOF cases were explained, guiding students in analysis and discussion. Students were organized into four teams under the instructors’ supervision, fostering cooperative learning and communication, thereby deepening their understanding of the disease through continuous inquiry and exploration (Fig.  2 L). In the routinely equipped PBL classroom with standard heart models (Fig.  2 J, K), all students had prior exposure to human anatomy and were familiar with these models. Both groups were provided with four standard heart models for reference, while the CBL-3DP group received additional four 3D-printed models depicting TOF anomalies, enriching their learning experience (Fig.  2 D, G). After the lesson, summarization, and feedback sessions were conducted to consolidate group discussions’ outcomes, evaluate teaching effectiveness, and assess learning outcomes.

Heart models utilized in instructional sessions. A External perspective of 3D digital models. B, C Cross-sectional views following trans-septal sagittal dissection of the 3D digital model (PS: Pulmonary Stenosis; OA: Overriding Aorta; VSD: Ventricular Septal Defect; RVH: Right Ventricular Hypertrophy). D External depiction of rigid 3D printed model. E, F Sagittal sections of the rigid 3D printed model. G External portrayal of flexible 3D printed model. H, I Sagittal sections of the flexible 3D printed model. J, K The normal heart model employed in the instruction of the CBL group. L Ongoing classroom session

Teaching effectiveness assessment

Following the instructional session, participants from the two groups underwent a theoretical examination to assess their comprehension of the taught material. This assessment covered domains such as pathological anatomy, clinical manifestations, imaging data interpretation, diagnosis, and treatment relevant to TOF. Additionally, structured questionnaires were administered to evaluate the efficacy of the pedagogical approach employed. The questionnaire consisted of six questions designed to gauge participants’ understanding of the teaching content, enhancement of diagnostic skills, cultivation of critical thinking and clinical reasoning abilities, bolstering of confidence in managing TOF cases, satisfaction with the teaching mode, and satisfaction with the CBL methodology.

The questionnaire employed a 5-point Likert scale to gauge responses, with 5 indicating “strongly satisfied/agree,” 4 for “satisfied/agree,” 3 denoting “neutral,” 2 reflecting “dissatisfied/disagree,” and 1 indicating “strongly dissatisfied/disagree.” It comprised six questions, with the initial two probing participants’ knowledge acquisition, questions 3 and 4 exploring satisfaction regarding enhanced competence, and the final two assessing satisfaction with teaching methods and modes. Additionally, participants were encouraged to provide suggestions at the end of the questionnaire. To ensure the questionnaire’s validity, five esteemed lecturers in basic medical sciences with more than 10 years of experience verified its content and assessed its Content Validity Ratio and Content Validity Index to ensure alignment with the study’s objectives.

Statistical analysis

Statistical analyses were conducted utilizing GraphPad Prism 9.0 software. Aggregate score data for both groups were presented as mean ± standard deviation (x ± s). The gender comparisons were analyzed with the chi-square (χ2) test, while the other variables were compared using the Mann-Whitney U test. The threshold for determining statistical significance was set at P  < 0.05.

Three-dimensional printing models

After configuring the structural colors of each component (Fig.  2 A, B, C), we printed four color TOF models using both rigid and flexible materials, resulting in four life-sized TOF models. Two color TOF models were created using rigid materials (Fig.  2 D, E, F). These models, exhibiting resistance to deformation, and with a firm texture, smooth and glossy surface, and good transparency, allowing visibility of the internal structures, were deemed conducive to teaching and observation. We also fabricated two color TOF models using flexible materials (Fig.  2 G, H, I), characterized by soft texture, opacity, and deformability, allowing for easy manipulation and cutting. It has potential utility beyond observational purposes. It can serve as a valuable tool for simulating surgical interventions and may be employed to create tomographic anatomical specimens. In this study, both material models were suitable for observation in the classroom. The participants were able to discern the four pathological changes characteristic of TOF from surface examination or cross-sectional analysis.

Baseline characteristics of the students

In total, 60 students were included in this study. The CBL group comprised 30 students (14 males and 16 females), with an average age of (21.20 ± 0.76) years. The CBL-3DP group consisted of 30 students (17 males and 13 females) with an average age of 20.96 years. All the students completed the study procedures. There were no significant differences in age, sex ratio, or pre-class exam scores between the two groups ( P  > 0.05), indicating that the baseline scores between the two groups were comparable (Table  1 ).

Theoretical examination results

All students completed the research procedures as planned. The post-class theoretical examination encompassed assessment of pathological anatomy, clinical presentations, imaging data interpretation, diagnosis, and treatment pertinent to TOF. Notably, no statistically significant disparities were observed in the scores on clinical manifestations, diagnosis and treatment components between the cohorts, as delineated in Table  2 . Conversely, discernible distinctions were evident whereby the CBL-3DP group outperformed the CBL group notably in pathological anatomy, imaging data interpretation, and overall aggregate scores ( P  < 0.05).

Results of the questionnaires

All the 60 participants submitted the questionnaire. Comparing the CBL and CBL-3DP groups, the scores from the CBL-3DP group showed significant improvements in many areas. This included satisfaction with the teaching mode, promotion of diagnostic skills, bolstering of self-assurance in managing TOF cases, and cultivation of critical thinking and clinical reasoning abilities (Fig.  3 B, C, D, E). All of which improved significantly ( P  < 0.05 for the first aspects and P  < 0.01 for the rest). However, the two groups were not comparable ( P  > 0.05) in terms of understanding of the teaching content and Satisfaction with the CBL methodology (Fig.  3 A, F).

Upon completion of the questionnaires, participants were invited to proffer recommendations. Notably, in the CBL group, seven students expressed challenges in comprehending TOF and indicated a need for additional time for consolidation to enhance understanding. Conversely, within the CBL-3DP group, twelve students advocated for the augmentation of model repertoire and the expansion of disease-related data collection to bolster pedagogical efficacy across other didactic domains.

Five-point Likert scores of students’ attitudes in CBL ( n  = 30) and CBL-3DP ( n  = 30) groups. A Understanding of teaching content. B Promotion of diagnostic skills. C Cultivation of critical thinking and clinical reasoning abilities. D Bolstering of self-assurance in managing TOF cases. E Satisfaction with the teaching mode. F Satisfaction with the CBL methodology. ns No significant difference, * p  < 0.05, ** p  < 0.01, *** p  < 0.001

TOF presents a significant challenge in clinical practice, necessitating a comprehensive understanding for effective diagnosis and treatment [ 12 ]. Traditional teaching methods in medical schools have relied on conventional resources such as textbooks, 2D illustrations, cadaver dissections, and radiographic materials to impart knowledge about complex conditions like TOF [ 13 ]. However, the limitations of these methods in fully engaging students and bridging the gap between theoretical knowledge and practical application have prompted a need for innovative instructional approaches.

CBL has emerged as a valuable tool in medical education, offering students opportunities to engage with authentic clinical cases through group discussions and inquiry-based learning [ 14 ]. By actively involving students in problem-solving and decision-making processes, CBL facilitates the application of theoretical knowledge to real-world scenarios, thus better-preparing students for future clinical practice [ 15 ]. Our investigation revealed that both groups of students exhibited comparable levels of satisfaction with the CBL methodology, devoid of discernible disparities.

CHD presents a formidable challenge due to the intricate nature of anatomical anomalies, the diverse spectrum of conditions, and individual variations [ 16 ]. Utilizing 3D-printed physical models, derived from patient imaging data, can significantly enhance comprehension of complex anatomical structures [ 17 ]. These models have proven invaluable in guiding surgical planning, providing training for junior or inexperienced pediatric residents, and educating healthcare professionals and parents of patients [ 18 ]. Studies indicate that as much as 50% of pediatric surgical decisions can be influenced by the insights gained from 3D printed models [ 19 ]. By providing tangible, anatomically accurate models, 3D printing offers a unique opportunity for people to visualize complex structures and enhance their understanding of anatomical intricacies. Our study utilized full-color, to-scale 3D printed models to illustrate the structural abnormalities associated with TOF, thereby enriching classroom sessions and facilitating a deeper comprehension of the condition.

Comparative analysis between the CBL-3DP group and the CBL group revealed significant improvements in post-test performance, particularly in pathological anatomy and imaging data interpretation. Additionally, questionnaire responses indicated higher levels of satisfaction and confidence among students in the CBL-3DP group, highlighting the positive impact of incorporating 3D printed models into the learning environment, improving the effectiveness of CBL classroom instruction. Despite the merits, our study has limitations. Primarily, participants were exclusively drawn from the same grade level within a single college, possibly engendering bias owing to shared learning backgrounds. Future research could further strengthen these findings by expanding the sample size and including long-term follow-up to assess the retention of knowledge and skills. Additionally, the influence of the 3D models depicting a normal heart on the learning process and its potential to introduce bias into the results warrants consideration, highlighting a need for scrutiny in subsequent studies.

As medical science continues to advance, the need for effective teaching methods becomes increasingly paramount. Our study underscores the potential of combining active learning approaches like CBL with innovative technologies such as 3D printing to enhance teaching effectiveness, improve knowledge acquisition, and foster students’ confidence and enthusiasm in pursuing clinical careers. Moving forward, further research and integration of such methodologies are essential for meeting the evolving demands of medical education, especially in areas involving complex anatomical understanding.

Conclusions

Integrating 3D-printed models with the CBL method is feasible and effective in TOF instruction. The demonstrated success of this method warrants broad implementation in medical education, particularly for complex anatomical topics.

Data availability

All data supporting the conclusions of this research are available upon reasonable request from the corresponding author.

Apitz C, Webb GD, Redington AN. Tetralogy of Fallot. Lancet. 2009;374:1462–71.

Article   Google Scholar  

Ghosh RM, Jolley MA, Mascio CE, Chen JM, Fuller S, Rome JJ, et al. Clinical 3D modeling to guide pediatric cardiothoracic surgery and intervention using 3D printed anatomic models, computer aided design and virtual reality. 3D Print Med. 2022;8:11.

Chakrabarti R, Wardle K, Wright T, Bennie T, Gishen F. Approaching an undergraduate medical curriculum map: challenges and expectations. BMC Med Educ. 2021;21:341.

Donkin R, Yule H, Fyfe T. Online case-based learning in medical education: a scoping review. BMC Med Educ. 2023;23:564.

Novack JP. Designing cases for case-based immunology teaching in large medical school classes. Front Immunol. 2020;11:995.

Chen HC, Van Den Broek WES, Ten Cate O. The case for use of entrustable professional activities in undergraduate medical education. Acad Med. 2015;90:431–6.

Wang M, Sun Z, Jia M, Wang Y, Wang H, Zhu X, et al. Intelligent virtual case learning system based on real medical records and natural language processing. BMC Med Inf Decis Mak. 2022;22:60.

Yoo S-J, Thabit O, Kim EK, Ide H, Yim D, Dragulescu A, et al. 3D printing in medicine of congenital heart diseases. 3D Print Med. 2015;2:3.

Yammine K, Violato C. A meta-analysis of the educational effectiveness of three-dimensional visualization technologies in teaching anatomy. Anat Sci Educ. 2015;8:525–38.

Miao H, Ding J, Gong X, Zhao J, Li H, Xiong K, et al. Application of 3D-printed pulmonary segment specimens in experimental teaching of sectional anatomy. BMC Surg. 2023;23:109.

Sun Z, Wong YH, Yeong CH. Patient-specific 3D-printed low-cost models in medical education and clinical practice. Micromachines (Basel). 2023;14:464.

Downing TE, Kim YY. Tetralogy of Fallot: general principles of management. Cardiol Clin. 2015;33:531–41. vii–viii.

Jia X, Zeng W, Zhang Q. Combined administration of problem- and lecture-based learning teaching models in medical education in China: a meta-analysis of randomized controlled trials. Med (Baltim). 2018;97:e11366.

McLean SF. Case-based learning and its application in medical and health-care fields: a review of worldwide literature. J Med Educ Curric Dev. 2016;3:JMECD.S20377.

Zeng N, Lu H, Li S, Yang Q, Liu F, Pan H, et al. Application of the combination of CBL teaching method and SEGUE framework to improve the doctor-patient communication skills of resident physicians in otolaryngology department. Bmc Med Educ. 2024;24:201.

Sun Z. Patient-specific 3D-printed models in pediatric congenital heart disease. Children. 2023;10:319.

Meyer-Szary J, Luis MS, Mikulski S, Patel A, Schulz F, Tretiakow D, et al. The role of 3D printing in planning complex medical procedures and training of medical professionals—cross-sectional multispecialty review. IJERPH. 2022;19:3331.

Sun Z, Wee C. 3D printed models in cardiovascular disease: an exciting future to deliver personalized medicine. Micromachines-basel. 2022;13:1575.

Valverde I, Gomez-Ciriza G, Hussain T, Suarez-Mejias C, Velasco-Forte MN, Byrne N, et al. Three-dimensional printed models for surgical planning of complex congenital heart defects: an international multicentre study. Eur J Cardio-thorac. 2017;52:1139–48.

Download references

Acknowledgements

We extend our sincere appreciation to the instructors and students whose invaluable participated in this study.

This paper received support from the Education Department of Anhui Province, China (Grant Numbers 2022jyxm1693, 2022jyxm1694, 2022xskc103, 2018jyxm1280).

Author information

Jian Zhao and Xin Gong are joint first authors.

Authors and Affiliations

Department of Human Anatomy, Wannan Medical College, No.22 West Wenchang Road, Wuhu, 241002, China

Jian Zhao, Xin Gong, Jian Ding, Rui Huang & Huachun Miao

Department of Cardio-Thoracic Surgery, Yijishan Hospital of Wannan Medical College, Wuhu, China

Kepin Xiong

Zhuhai Sailner 3D Technology Co., Ltd., Zhuhai, China

Kangle Zhuang

School of Basic Medical Sciences, Wannan Medical College, Wuhu, China

You can also search for this author in PubMed   Google Scholar

Contributions

Jian Zhao and Huachun Miao designed the research. Jian Zhao, Xin Gong, Jian Ding, Kepin Xiong designed the tests and questionnaires. Kangle Zhuang processed the imaging data and printed the models. Xing Gong and Kepin Xiong implemented the teaching. Jian Zhao and Rui Huang collected the data and performed the statistical analysis. Jian Zhao and Huachun Miao prepared the manuscript. Shu Li and Huachun Miao revised the manuscript. Shu Li provided the Funding acquisition. All authors reviewed and approved the final manuscript.

Corresponding authors

Correspondence to Shu Li or Huachun Miao .

Ethics declarations

Ethics approval and consent to participate.

This investigation received ethical approval from the Ethical Committee of School of Basic Medical Sciences, Wannan Medical College (ECBMSWMC2022-1-12). All methodologies adhered strictly to established protocols and guidelines. Written informed consent was obtained from the study participants to take part in the study.

Consent for publication

Written informed consent was obtained from the individuals for the publication of any potentially identifiable images or data included in this article.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, supplementary material 3, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Zhao, J., Gong, X., Ding, J. et al. Integration of case-based learning and three-dimensional printing for tetralogy of fallot instruction in clinical medical undergraduates: a randomized controlled trial. BMC Med Educ 24 , 571 (2024). https://doi.org/10.1186/s12909-024-05583-z

Download citation

Received : 03 March 2024

Accepted : 21 May 2024

Published : 24 May 2024

DOI : https://doi.org/10.1186/s12909-024-05583-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Medical education
• Case-based learning
• 3D printing
• Tetralogy of fallot
• Medical undergraduates

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]