program case study

How to write a case study — examples, templates, and tools

It’s a marketer’s job to communicate the effectiveness of a product or service to potential and current customers to convince them to buy and keep business moving. One of the best methods for doing this is to share success stories that are relatable to prospects and customers based on their pain points, experiences, and overall needs.

That’s where case studies come in. Case studies are an essential part of a content marketing plan. These in-depth stories of customer experiences are some of the most effective at demonstrating the value of a product or service. Yet many marketers don’t use them, whether because of their regimented formats or the process of customer involvement and approval.

A case study is a powerful tool for showcasing your hard work and the success your customer achieved. But writing a great case study can be difficult if you’ve never done it before or if it’s been a while. This guide will show you how to write an effective case study and provide real-world examples and templates that will keep readers engaged and support your business.

In this article, you’ll learn:

What is a case study?

How to write a case study, case study templates, case study examples, case study tools.

A case study is the detailed story of a customer’s experience with a product or service that demonstrates their success and often includes measurable outcomes. Case studies are used in a range of fields and for various reasons, from business to academic research. They’re especially impactful in marketing as brands work to convince and convert consumers with relatable, real-world stories of actual customer experiences.

The best case studies tell the story of a customer’s success, including the steps they took, the results they achieved, and the support they received from a brand along the way. To write a great case study, you need to:

• Celebrate the customer and make them — not a product or service — the star of the story.
• Craft the story with specific audiences or target segments in mind so that the story of one customer will be viewed as relatable and actionable for another customer.
• Write copy that is easy to read and engaging so that readers will gain the insights and messages intended.
• Follow a standardized format that includes all of the essentials a potential customer would find interesting and useful.
• Support all of the claims for success made in the story with data in the forms of hard numbers and customer statements.

Case studies are a type of review but more in depth, aiming to show — rather than just tell — the positive experiences that customers have with a brand. Notably, 89% of consumers read reviews before deciding to buy, and 79% view case study content as part of their purchasing process. When it comes to B2B sales, 52% of buyers rank case studies as an important part of their evaluation process.

Telling a brand story through the experience of a tried-and-true customer matters. The story is relatable to potential new customers as they imagine themselves in the shoes of the company or individual featured in the case study. Showcasing previous customers can help new ones see themselves engaging with your brand in the ways that are most meaningful to them.

Besides sharing the perspective of another customer, case studies stand out from other content marketing forms because they are based on evidence. Whether pulling from client testimonials or data-driven results, case studies tend to have more impact on new business because the story contains information that is both objective (data) and subjective (customer experience) — and the brand doesn’t sound too self-promotional.

Case studies are unique in that there’s a fairly standardized format for telling a customer’s story. But that doesn’t mean there isn’t room for creativity. It’s all about making sure that teams are clear on the goals for the case study — along with strategies for supporting content and channels — and understanding how the story fits within the framework of the company’s overall marketing goals.

Here are the basic steps to writing a good case study.

1. Identify your goal

Start by defining exactly who your case study will be designed to help. Case studies are about specific instances where a company works with a customer to achieve a goal. Identify which customers are likely to have these goals, as well as other needs the story should cover to appeal to them.

The answer is often found in one of the buyer personas that have been constructed as part of your larger marketing strategy. This can include anything from new leads generated by the marketing team to long-term customers that are being pressed for cross-sell opportunities. In all of these cases, demonstrating value through a relatable customer success story can be part of the solution to conversion.

2. Choose your client or subject

Who you highlight matters. Case studies tie brands together that might otherwise not cross paths. A writer will want to ensure that the highlighted customer aligns with their own company’s brand identity and offerings. Look for a customer with positive name recognition who has had great success with a product or service and is willing to be an advocate.

The client should also match up with the identified target audience. Whichever company or individual is selected should be a reflection of other potential customers who can see themselves in similar circumstances, having the same problems and possible solutions.

Some of the most compelling case studies feature customers who:

• Switch from one product or service to another while naming competitors that missed the mark.
• Experience measurable results that are relatable to others in a specific industry.
• Represent well-known brands and recognizable names that are likely to compel action.
• Advocate for a product or service as a champion and are well-versed in its advantages.

Whoever or whatever customer is selected, marketers must ensure they have the permission of the company involved before getting started. Some brands have strict review and approval procedures for any official marketing or promotional materials that include their name. Acquiring those approvals in advance will prevent any miscommunication or wasted effort if there is an issue with their legal or compliance teams.

3. Conduct research and compile data

Substantiating the claims made in a case study — either by the marketing team or customers themselves — adds validity to the story. To do this, include data and feedback from the client that defines what success looks like. This can be anything from demonstrating return on investment (ROI) to a specific metric the customer was striving to improve. Case studies should prove how an outcome was achieved and show tangible results that indicate to the customer that your solution is the right one.

This step could also include customer interviews. Make sure that the people being interviewed are key stakeholders in the purchase decision or deployment and use of the product or service that is being highlighted. Content writers should work off a set list of questions prepared in advance. It can be helpful to share these with the interviewees beforehand so they have time to consider and craft their responses. One of the best interview tactics to keep in mind is to ask questions where yes and no are not natural answers. This way, your subject will provide more open-ended responses that produce more meaningful content.

4. Choose the right format

There are a number of different ways to format a case study. Depending on what you hope to achieve, one style will be better than another. However, there are some common elements to include, such as:

• An engaging headline
• A subject and customer introduction
• The unique challenge or challenges the customer faced
• The solution the customer used to solve the problem
• The results achieved
• Data and statistics to back up claims of success
• A strong call to action (CTA) to engage with the vendor

It’s also important to note that while case studies are traditionally written as stories, they don’t have to be in a written format. Some companies choose to get more creative with their case studies and produce multimedia content, depending on their audience and objectives. Case study formats can include traditional print stories, interactive web or social content, data-heavy infographics, professionally shot videos, podcasts, and more.

5. Write your case study

We’ll go into more detail later about how exactly to write a case study, including templates and examples. Generally speaking, though, there are a few things to keep in mind when writing your case study.

• Be clear and concise. Readers want to get to the point of the story quickly and easily, and they’ll be looking to see themselves reflected in the story right from the start.
• Provide a big picture. Always make sure to explain who the client is, their goals, and how they achieved success in a short introduction to engage the reader.
• Construct a clear narrative. Stick to the story from the perspective of the customer and what they needed to solve instead of just listing product features or benefits.
• Leverage graphics. Incorporating infographics, charts, and sidebars can be a more engaging and eye-catching way to share key statistics and data in readable ways.
• Offer the right amount of detail. Most case studies are one or two pages with clear sections that a reader can skim to find the information most important to them.
• Include data to support claims. Show real results — both facts and figures and customer quotes — to demonstrate credibility and prove the solution works.

6. Promote your story

Marketers have a number of options for distribution of a freshly minted case study. Many brands choose to publish case studies on their website and post them on social media. This can help support SEO and organic content strategies while also boosting company credibility and trust as visitors see that other businesses have used the product or service.

Marketers are always looking for quality content they can use for lead generation. Consider offering a case study as gated content behind a form on a landing page or as an offer in an email message. One great way to do this is to summarize the content and tease the full story available for download after the user takes an action.

Sales teams can also leverage case studies, so be sure they are aware that the assets exist once they’re published. Especially when it comes to larger B2B sales, companies often ask for examples of similar customer challenges that have been solved.

Now that you’ve learned a bit about case studies and what they should include, you may be wondering how to start creating great customer story content. Here are a couple of templates you can use to structure your case study.

Template 1 — Challenge-solution-result format

• Start with an engaging title. This should be fewer than 70 characters long for SEO best practices. One of the best ways to approach the title is to include the customer’s name and a hint at the challenge they overcame in the end.
• Create an introduction. Lead with an explanation as to who the customer is, the need they had, and the opportunity they found with a specific product or solution. Writers can also suggest the success the customer experienced with the solution they chose.
• Present the challenge. This should be several paragraphs long and explain the problem the customer faced and the issues they were trying to solve. Details should tie into the company’s products and services naturally. This section needs to be the most relatable to the reader so they can picture themselves in a similar situation.
• Share the solution. Explain which product or service offered was the ideal fit for the customer and why. Feel free to delve into their experience setting up, purchasing, and onboarding the solution.
• Explain the results. Demonstrate the impact of the solution they chose by backing up their positive experience with data. Fill in with customer quotes and tangible, measurable results that show the effect of their choice.
• Ask for action. Include a CTA at the end of the case study that invites readers to reach out for more information, try a demo, or learn more — to nurture them further in the marketing pipeline. What you ask of the reader should tie directly into the goals that were established for the case study in the first place.

Template 2 — Data-driven format

• Start with an engaging title. Be sure to include a statistic or data point in the first 70 characters. Again, it’s best to include the customer’s name as part of the title.
• Create an overview. Share the customer’s background and a short version of the challenge they faced. Present the reason a particular product or service was chosen, and feel free to include quotes from the customer about their selection process.
• Present data point 1. Isolate the first metric that the customer used to define success and explain how the product or solution helped to achieve this goal. Provide data points and quotes to substantiate the claim that success was achieved.
• Present data point 2. Isolate the second metric that the customer used to define success and explain what the product or solution did to achieve this goal. Provide data points and quotes to substantiate the claim that success was achieved.
• Present data point 3. Isolate the final metric that the customer used to define success and explain what the product or solution did to achieve this goal. Provide data points and quotes to substantiate the claim that success was achieved.
• Summarize the results. Reiterate the fact that the customer was able to achieve success thanks to a specific product or service. Include quotes and statements that reflect customer satisfaction and suggest they plan to continue using the solution.
• Ask for action. Include a CTA at the end of the case study that asks readers to reach out for more information, try a demo, or learn more — to further nurture them in the marketing pipeline. Again, remember that this is where marketers can look to convert their content into action with the customer.

While templates are helpful, seeing a case study in action can also be a great way to learn. Here are some examples of how Adobe customers have experienced success.

Juniper Networks

One example is the Adobe and Juniper Networks case study , which puts the reader in the customer’s shoes. The beginning of the story quickly orients the reader so that they know exactly who the article is about and what they were trying to achieve. Solutions are outlined in a way that shows Adobe Experience Manager is the best choice and a natural fit for the customer. Along the way, quotes from the client are incorporated to help add validity to the statements. The results in the case study are conveyed with clear evidence of scale and volume using tangible data.

The story of Lenovo’s journey with Adobe is one that spans years of planning, implementation, and rollout. The Lenovo case study does a great job of consolidating all of this into a relatable journey that other enterprise organizations can see themselves taking, despite the project size. This case study also features descriptive headers and compelling visual elements that engage the reader and strengthen the content.

Tata Consulting

When it comes to using data to show customer results, this case study does an excellent job of conveying details and numbers in an easy-to-digest manner. Bullet points at the start break up the content while also helping the reader understand exactly what the case study will be about. Tata Consulting used Adobe to deliver elevated, engaging content experiences for a large telecommunications client of its own — an objective that’s relatable for a lot of companies.

Case studies are a vital tool for any marketing team as they enable you to demonstrate the value of your company’s products and services to others. They help marketers do their job and add credibility to a brand trying to promote its solutions by using the experiences and stories of real customers.

When you’re ready to get started with a case study:

• Think about a few goals you’d like to accomplish with your content.
• Make a list of successful clients that would be strong candidates for a case study.
• Reach out to the client to get their approval and conduct an interview.
• Gather the data to present an engaging and effective customer story.

Adobe can help

There are several Adobe products that can help you craft compelling case studies. Adobe Experience Platform helps you collect data and deliver great customer experiences across every channel. Once you’ve created your case studies, Experience Platform will help you deliver the right information to the right customer at the right time for maximum impact.

To learn more, watch the Adobe Experience Platform story .

Keep in mind that the best case studies are backed by data. That’s where Adobe Real-Time Customer Data Platform and Adobe Analytics come into play. With Real-Time CDP, you can gather the data you need to build a great case study and target specific customers to deliver the content to the right audience at the perfect moment.

Watch the Real-Time CDP overview video to learn more.

Finally, Adobe Analytics turns real-time data into real-time insights. It helps your business collect and synthesize data from multiple platforms to make more informed decisions and create the best case study possible.

Request a demo to learn more about Adobe Analytics.

https://business.adobe.com/blog/perspectives/b2b-ecommerce-10-case-studies-inspire-you

https://business.adobe.com/blog/basics/business-case

https://business.adobe.com/blog/basics/what-is-real-time-analytics

The Compass for SBC

Helping you Implement Effective Social and Behavior Change Projects

CAPACITY STRENGTHENING TOOL

Home > All Tools > Using Case Studies to do Program Evaluation [Guide]

More Resources

Using case studies to do program evaluation [guide].

The purpose of a case study is to study intensely one set (or unit) of something—programs, cities, counties, worksites—as a distinct whole. A case study is a useful mechanism for evaluating programs, if it is used appropriately and if it is done well. A case study is particularly useful for evaluating unique programs, programs with unique outcomes, and programs carried out in turbulent or unpredictable environments. It is well suited to programs of which at least part of the object is to learn from the process.

As with any evaluation, the keys to effective use of a case study are to design the data collection effort with care, and to create a clear record of the methods used to perform the data collection and analysis. Also, as is true of any evaluation, the data must support the findings.

This guide includes instructions on:

• How to do a case study
• When to do a case study
• Data collection
• Data analysis and interpretation

Last modified: July 23, 2021

Language: English

• The Use of IVR to Support Monitoring and Evaluation of Media Interventions: A Case Study of the VOTO System in Rwanda

Share this Article

• Featured Collections

Case Pack: Program Evaluation

If you are like me, one of the key barriers in adopting cases is finding the right cases for your course. With this in mind, the HKS Case Program created this guide to simplify the process of identifying and selecting a case in the realm of program evaluation.

Cases help bring some of the course material to life and can be a great complement to other pedagogic approaches. For example, while the students in my course learn about different methods to evaluate the impact of the program, using the Jamaica PATH case has helped them more clearly see the advantages and disadvantages of these methods and, perhaps just as importantly, the tradeoffs that are often made in trying to select an evaluation design in the real world. 

I hope this guide is as useful to you as it has been to me. 

These short cases are designed to improve learning on a host of topics, from theory of change to impact evaluations, while underscoring the everyday tradeoffs organizations face when trying to do scientifically rigorous work in the practical constraints of the real world.

Theme I: Using Evidence

A. theory of change.

Case : Women as Leaders: Lessons from Political Quotas in India Length : 7 pages Learning Objective : Students learn how to assess the theory of change of an intervention aimed at improving political representation of women in India and also learn how to (and how not to) assess the causal effect of an intervention.

Case : Scared Straight: Freeport City Council Takes on Juvenile Delinquency Length : 4 pages Learning Objective : Students learn what is meant by "evidence" and what different forms of evidence they can draw on.

B. The Role of Evidence in Social Programs

Case : Providing Pensions for the Poor: Targeting Cash Transfers for the Elderly in Mexico Length : 5 pages Learning Objective : Introduces students to the use of evidence in social programs in the context of having to decide between 3 possible targeting mechanisms that the government of Mexico considered when launching a pension program for the elderly. Students are put in a position to confront some challenging tradeoffs between targeting accuracy and political, financial, and operational feasibility.

Theme II: Impact Evaluations

A. importance of evaluation design.

Video Case : New York City’s Teen ACTION Program: An Evaluation Gone Awry Length : 2 videos (total 10 minutes); 4 page background note. Learning Objective : Highlights the importance of good evaluation design, and raises key issues around statistical power, measurement error, internal validity, etc.

B. How to Design an Impact Evaluation (Non-Experimental Methods)

Case : Designing Impact Evaluations: Assessing Jamaica’s PATH Program Length : 4 pages Learning Objective : Students are asked to assess the strengths and weaknesses of three possible evaluation designs using as criteria the scientific rigor, political feasibility, logistical implications, and financial feasibility of each design. They are put in a position to confront the inevitable tradeoffs when trying to balance the scientific rigor of an evaluation design with the practical constraints of implementing it.

C. How to Design an Impact Evaluation (Experimental Methods)

Case : Devil in the Details: Designing a Social Impact Bond Agreement in Medellin Length : 15 pages Learning Objective : Under the challenge of crafting a Social Impact Bond to help reduce the teenage pregnancy rate in Medellin (Colombia), students are asked to grapple with key issues in designing an impact evaluation, including selecting treatment and control groups and identifying outcome measures that are both rigorous and practical to collect under the constraints the protagonists of the case face.

Theme III: Qualitative Analysis

Case : The Geography of Poverty: Exploring the Role of Neighborhoods in the Lives of Urban, Adolescent Poor Length : 5 pages Learning Objective : Students learn about the functions of qualitative research and closely examine how qualitative research can enhance the interpretation of quantitative data in the context of a mixed methods evaluation of Moving To Opportunity, a very ambitious anti-poverty program that consisted in offering low-income families in five U.S. cities (Baltimore, Boston, Chicago, Los Angeles and New York) the opportunity to move to neighborhoods with low levels of poverty.

Faculty Profile

Dan Levy , Senior Lecturer in Public Policy and Faculty Chair of the Kennedy School's SLATE Initiative , teaches courses in quantitative methods and program evaluation. He has served as a senior researcher at Mathematica Policy Research, a faculty affiliate at the Poverty Action Lab (MIT), and as consultant to several organizations including the World Bank.

Recent Cases by Dan

Betting Private Capital on Fixing Public Ills: Instiglio Brings Social Impact Bonds to Colombia Providing Pensions for the Poor: Targeting Cash Transfers for the Elderly in Mexico Devil in the Details: Designing a Social Impact Bond Agreement in Medellin

Organizing Your Social Sciences Research Assignments

• Annotated Bibliography
• Analyzing a Scholarly Journal Article
• Group Presentations
• Dealing with Nervousness
• Using Visual Aids
• Grading Someone Else's Paper
• Types of Structured Group Activities
• Group Project Survival Skills
• Leading a Class Discussion
• Multiple Book Review Essay
• Reviewing Collected Works
• Writing a Case Analysis Paper
• Writing a Case Study
• About Informed Consent
• Writing Field Notes
• Writing a Policy Memo
• Writing a Reflective Paper
• Writing a Research Proposal
• Generative AI and Writing
• Acknowledgments

A case study research paper examines a person, place, event, condition, phenomenon, or other type of subject of analysis in order to extrapolate  key themes and results that help predict future trends, illuminate previously hidden issues that can be applied to practice, and/or provide a means for understanding an important research problem with greater clarity. A case study research paper usually examines a single subject of analysis, but case study papers can also be designed as a comparative investigation that shows relationships between two or more subjects. The methods used to study a case can rest within a quantitative, qualitative, or mixed-method investigative paradigm.

Case Studies. Writing@CSU. Colorado State University; Mills, Albert J. , Gabrielle Durepos, and Eiden Wiebe, editors. Encyclopedia of Case Study Research . Thousand Oaks, CA: SAGE Publications, 2010 ; “What is a Case Study?” In Swanborn, Peter G. Case Study Research: What, Why and How? London: SAGE, 2010.

How to Approach Writing a Case Study Research Paper

General information about how to choose a topic to investigate can be found under the " Choosing a Research Problem " tab in the Organizing Your Social Sciences Research Paper writing guide. Review this page because it may help you identify a subject of analysis that can be investigated using a case study design.

However, identifying a case to investigate involves more than choosing the research problem . A case study encompasses a problem contextualized around the application of in-depth analysis, interpretation, and discussion, often resulting in specific recommendations for action or for improving existing conditions. As Seawright and Gerring note, practical considerations such as time and access to information can influence case selection, but these issues should not be the sole factors used in describing the methodological justification for identifying a particular case to study. Given this, selecting a case includes considering the following:

• The case represents an unusual or atypical example of a research problem that requires more in-depth analysis? Cases often represent a topic that rests on the fringes of prior investigations because the case may provide new ways of understanding the research problem. For example, if the research problem is to identify strategies to improve policies that support girl's access to secondary education in predominantly Muslim nations, you could consider using Azerbaijan as a case study rather than selecting a more obvious nation in the Middle East. Doing so may reveal important new insights into recommending how governments in other predominantly Muslim nations can formulate policies that support improved access to education for girls.
• The case provides important insight or illuminate a previously hidden problem? In-depth analysis of a case can be based on the hypothesis that the case study will reveal trends or issues that have not been exposed in prior research or will reveal new and important implications for practice. For example, anecdotal evidence may suggest drug use among homeless veterans is related to their patterns of travel throughout the day. Assuming prior studies have not looked at individual travel choices as a way to study access to illicit drug use, a case study that observes a homeless veteran could reveal how issues of personal mobility choices facilitate regular access to illicit drugs. Note that it is important to conduct a thorough literature review to ensure that your assumption about the need to reveal new insights or previously hidden problems is valid and evidence-based.
• The case challenges and offers a counter-point to prevailing assumptions? Over time, research on any given topic can fall into a trap of developing assumptions based on outdated studies that are still applied to new or changing conditions or the idea that something should simply be accepted as "common sense," even though the issue has not been thoroughly tested in current practice. A case study analysis may offer an opportunity to gather evidence that challenges prevailing assumptions about a research problem and provide a new set of recommendations applied to practice that have not been tested previously. For example, perhaps there has been a long practice among scholars to apply a particular theory in explaining the relationship between two subjects of analysis. Your case could challenge this assumption by applying an innovative theoretical framework [perhaps borrowed from another discipline] to explore whether this approach offers new ways of understanding the research problem. Taking a contrarian stance is one of the most important ways that new knowledge and understanding develops from existing literature.
• The case provides an opportunity to pursue action leading to the resolution of a problem? Another way to think about choosing a case to study is to consider how the results from investigating a particular case may result in findings that reveal ways in which to resolve an existing or emerging problem. For example, studying the case of an unforeseen incident, such as a fatal accident at a railroad crossing, can reveal hidden issues that could be applied to preventative measures that contribute to reducing the chance of accidents in the future. In this example, a case study investigating the accident could lead to a better understanding of where to strategically locate additional signals at other railroad crossings so as to better warn drivers of an approaching train, particularly when visibility is hindered by heavy rain, fog, or at night.
• The case offers a new direction in future research? A case study can be used as a tool for an exploratory investigation that highlights the need for further research about the problem. A case can be used when there are few studies that help predict an outcome or that establish a clear understanding about how best to proceed in addressing a problem. For example, after conducting a thorough literature review [very important!], you discover that little research exists showing the ways in which women contribute to promoting water conservation in rural communities of east central Africa. A case study of how women contribute to saving water in a rural village of Uganda can lay the foundation for understanding the need for more thorough research that documents how women in their roles as cooks and family caregivers think about water as a valuable resource within their community. This example of a case study could also point to the need for scholars to build new theoretical frameworks around the topic [e.g., applying feminist theories of work and family to the issue of water conservation].

Eisenhardt, Kathleen M. “Building Theories from Case Study Research.” Academy of Management Review 14 (October 1989): 532-550; Emmel, Nick. Sampling and Choosing Cases in Qualitative Research: A Realist Approach . Thousand Oaks, CA: SAGE Publications, 2013; Gerring, John. “What Is a Case Study and What Is It Good for?” American Political Science Review 98 (May 2004): 341-354; Mills, Albert J. , Gabrielle Durepos, and Eiden Wiebe, editors. Encyclopedia of Case Study Research . Thousand Oaks, CA: SAGE Publications, 2010; Seawright, Jason and John Gerring. "Case Selection Techniques in Case Study Research." Political Research Quarterly 61 (June 2008): 294-308.

Structure and Writing Style

The purpose of a paper in the social sciences designed around a case study is to thoroughly investigate a subject of analysis in order to reveal a new understanding about the research problem and, in so doing, contributing new knowledge to what is already known from previous studies. In applied social sciences disciplines [e.g., education, social work, public administration, etc.], case studies may also be used to reveal best practices, highlight key programs, or investigate interesting aspects of professional work.

In general, the structure of a case study research paper is not all that different from a standard college-level research paper. However, there are subtle differences you should be aware of. Here are the key elements to organizing and writing a case study research paper.

I.  Introduction

As with any research paper, your introduction should serve as a roadmap for your readers to ascertain the scope and purpose of your study . The introduction to a case study research paper, however, should not only describe the research problem and its significance, but you should also succinctly describe why the case is being used and how it relates to addressing the problem. The two elements should be linked. With this in mind, a good introduction answers these four questions:

• What is being studied? Describe the research problem and describe the subject of analysis [the case] you have chosen to address the problem. Explain how they are linked and what elements of the case will help to expand knowledge and understanding about the problem.
• Why is this topic important to investigate? Describe the significance of the research problem and state why a case study design and the subject of analysis that the paper is designed around is appropriate in addressing the problem.
• What did we know about this topic before I did this study? Provide background that helps lead the reader into the more in-depth literature review to follow. If applicable, summarize prior case study research applied to the research problem and why it fails to adequately address the problem. Describe why your case will be useful. If no prior case studies have been used to address the research problem, explain why you have selected this subject of analysis.
• How will this study advance new knowledge or new ways of understanding? Explain why your case study will be suitable in helping to expand knowledge and understanding about the research problem.

Each of these questions should be addressed in no more than a few paragraphs. Exceptions to this can be when you are addressing a complex research problem or subject of analysis that requires more in-depth background information.

II.  Literature Review

The literature review for a case study research paper is generally structured the same as it is for any college-level research paper. The difference, however, is that the literature review is focused on providing background information and  enabling historical interpretation of the subject of analysis in relation to the research problem the case is intended to address . This includes synthesizing studies that help to:

• Place relevant works in the context of their contribution to understanding the case study being investigated . This would involve summarizing studies that have used a similar subject of analysis to investigate the research problem. If there is literature using the same or a very similar case to study, you need to explain why duplicating past research is important [e.g., conditions have changed; prior studies were conducted long ago, etc.].
• Describe the relationship each work has to the others under consideration that informs the reader why this case is applicable . Your literature review should include a description of any works that support using the case to investigate the research problem and the underlying research questions.
• Identify new ways to interpret prior research using the case study . If applicable, review any research that has examined the research problem using a different research design. Explain how your use of a case study design may reveal new knowledge or a new perspective or that can redirect research in an important new direction.
• Resolve conflicts amongst seemingly contradictory previous studies . This refers to synthesizing any literature that points to unresolved issues of concern about the research problem and describing how the subject of analysis that forms the case study can help resolve these existing contradictions.
• Point the way in fulfilling a need for additional research . Your review should examine any literature that lays a foundation for understanding why your case study design and the subject of analysis around which you have designed your study may reveal a new way of approaching the research problem or offer a perspective that points to the need for additional research.
• Expose any gaps that exist in the literature that the case study could help to fill . Summarize any literature that not only shows how your subject of analysis contributes to understanding the research problem, but how your case contributes to a new way of understanding the problem that prior research has failed to do.
• Locate your own research within the context of existing literature [very important!] . Collectively, your literature review should always place your case study within the larger domain of prior research about the problem. The overarching purpose of reviewing pertinent literature in a case study paper is to demonstrate that you have thoroughly identified and synthesized prior studies in relation to explaining the relevance of the case in addressing the research problem.

III.  Method

In this section, you explain why you selected a particular case [i.e., subject of analysis] and the strategy you used to identify and ultimately decide that your case was appropriate in addressing the research problem. The way you describe the methods used varies depending on the type of subject of analysis that constitutes your case study.

If your subject of analysis is an incident or event . In the social and behavioral sciences, the event or incident that represents the case to be studied is usually bounded by time and place, with a clear beginning and end and with an identifiable location or position relative to its surroundings. The subject of analysis can be a rare or critical event or it can focus on a typical or regular event. The purpose of studying a rare event is to illuminate new ways of thinking about the broader research problem or to test a hypothesis. Critical incident case studies must describe the method by which you identified the event and explain the process by which you determined the validity of this case to inform broader perspectives about the research problem or to reveal new findings. However, the event does not have to be a rare or uniquely significant to support new thinking about the research problem or to challenge an existing hypothesis. For example, Walo, Bull, and Breen conducted a case study to identify and evaluate the direct and indirect economic benefits and costs of a local sports event in the City of Lismore, New South Wales, Australia. The purpose of their study was to provide new insights from measuring the impact of a typical local sports event that prior studies could not measure well because they focused on large "mega-events." Whether the event is rare or not, the methods section should include an explanation of the following characteristics of the event: a) when did it take place; b) what were the underlying circumstances leading to the event; and, c) what were the consequences of the event in relation to the research problem.

If your subject of analysis is a person. Explain why you selected this particular individual to be studied and describe what experiences they have had that provide an opportunity to advance new understandings about the research problem. Mention any background about this person which might help the reader understand the significance of their experiences that make them worthy of study. This includes describing the relationships this person has had with other people, institutions, and/or events that support using them as the subject for a case study research paper. It is particularly important to differentiate the person as the subject of analysis from others and to succinctly explain how the person relates to examining the research problem [e.g., why is one politician in a particular local election used to show an increase in voter turnout from any other candidate running in the election]. Note that these issues apply to a specific group of people used as a case study unit of analysis [e.g., a classroom of students].

If your subject of analysis is a place. In general, a case study that investigates a place suggests a subject of analysis that is unique or special in some way and that this uniqueness can be used to build new understanding or knowledge about the research problem. A case study of a place must not only describe its various attributes relevant to the research problem [e.g., physical, social, historical, cultural, economic, political], but you must state the method by which you determined that this place will illuminate new understandings about the research problem. It is also important to articulate why a particular place as the case for study is being used if similar places also exist [i.e., if you are studying patterns of homeless encampments of veterans in open spaces, explain why you are studying Echo Park in Los Angeles rather than Griffith Park?]. If applicable, describe what type of human activity involving this place makes it a good choice to study [e.g., prior research suggests Echo Park has more homeless veterans].

If your subject of analysis is a phenomenon. A phenomenon refers to a fact, occurrence, or circumstance that can be studied or observed but with the cause or explanation to be in question. In this sense, a phenomenon that forms your subject of analysis can encompass anything that can be observed or presumed to exist but is not fully understood. In the social and behavioral sciences, the case usually focuses on human interaction within a complex physical, social, economic, cultural, or political system. For example, the phenomenon could be the observation that many vehicles used by ISIS fighters are small trucks with English language advertisements on them. The research problem could be that ISIS fighters are difficult to combat because they are highly mobile. The research questions could be how and by what means are these vehicles used by ISIS being supplied to the militants and how might supply lines to these vehicles be cut off? How might knowing the suppliers of these trucks reveal larger networks of collaborators and financial support? A case study of a phenomenon most often encompasses an in-depth analysis of a cause and effect that is grounded in an interactive relationship between people and their environment in some way.

NOTE:   The choice of the case or set of cases to study cannot appear random. Evidence that supports the method by which you identified and chose your subject of analysis should clearly support investigation of the research problem and linked to key findings from your literature review. Be sure to cite any studies that helped you determine that the case you chose was appropriate for examining the problem.

IV.  Discussion

The main elements of your discussion section are generally the same as any research paper, but centered around interpreting and drawing conclusions about the key findings from your analysis of the case study. Note that a general social sciences research paper may contain a separate section to report findings. However, in a paper designed around a case study, it is common to combine a description of the results with the discussion about their implications. The objectives of your discussion section should include the following:

Reiterate the Research Problem/State the Major Findings Briefly reiterate the research problem you are investigating and explain why the subject of analysis around which you designed the case study were used. You should then describe the findings revealed from your study of the case using direct, declarative, and succinct proclamation of the study results. Highlight any findings that were unexpected or especially profound.

Explain the Meaning of the Findings and Why They are Important Systematically explain the meaning of your case study findings and why you believe they are important. Begin this part of the section by repeating what you consider to be your most important or surprising finding first, then systematically review each finding. Be sure to thoroughly extrapolate what your analysis of the case can tell the reader about situations or conditions beyond the actual case that was studied while, at the same time, being careful not to misconstrue or conflate a finding that undermines the external validity of your conclusions.

Relate the Findings to Similar Studies No study in the social sciences is so novel or possesses such a restricted focus that it has absolutely no relation to previously published research. The discussion section should relate your case study results to those found in other studies, particularly if questions raised from prior studies served as the motivation for choosing your subject of analysis. This is important because comparing and contrasting the findings of other studies helps support the overall importance of your results and it highlights how and in what ways your case study design and the subject of analysis differs from prior research about the topic.

Consider Alternative Explanations of the Findings Remember that the purpose of social science research is to discover and not to prove. When writing the discussion section, you should carefully consider all possible explanations revealed by the case study results, rather than just those that fit your hypothesis or prior assumptions and biases. Be alert to what the in-depth analysis of the case may reveal about the research problem, including offering a contrarian perspective to what scholars have stated in prior research if that is how the findings can be interpreted from your case.

Acknowledge the Study's Limitations You can state the study's limitations in the conclusion section of your paper but describing the limitations of your subject of analysis in the discussion section provides an opportunity to identify the limitations and explain why they are not significant. This part of the discussion section should also note any unanswered questions or issues your case study could not address. More detailed information about how to document any limitations to your research can be found here .

Suggest Areas for Further Research Although your case study may offer important insights about the research problem, there are likely additional questions related to the problem that remain unanswered or findings that unexpectedly revealed themselves as a result of your in-depth analysis of the case. Be sure that the recommendations for further research are linked to the research problem and that you explain why your recommendations are valid in other contexts and based on the original assumptions of your study.

V.  Conclusion

As with any research paper, you should summarize your conclusion in clear, simple language; emphasize how the findings from your case study differs from or supports prior research and why. Do not simply reiterate the discussion section. Provide a synthesis of key findings presented in the paper to show how these converge to address the research problem. If you haven't already done so in the discussion section, be sure to document the limitations of your case study and any need for further research.

The function of your paper's conclusion is to: 1) reiterate the main argument supported by the findings from your case study; 2) state clearly the context, background, and necessity of pursuing the research problem using a case study design in relation to an issue, controversy, or a gap found from reviewing the literature; and, 3) provide a place to persuasively and succinctly restate the significance of your research problem, given that the reader has now been presented with in-depth information about the topic.

Consider the following points to help ensure your conclusion is appropriate:

• If the argument or purpose of your paper is complex, you may need to summarize these points for your reader.
• If prior to your conclusion, you have not yet explained the significance of your findings or if you are proceeding inductively, use the conclusion of your paper to describe your main points and explain their significance.
• Move from a detailed to a general level of consideration of the case study's findings that returns the topic to the context provided by the introduction or within a new context that emerges from your case study findings.

Note that, depending on the discipline you are writing in or the preferences of your professor, the concluding paragraph may contain your final reflections on the evidence presented as it applies to practice or on the essay's central research problem. However, the nature of being introspective about the subject of analysis you have investigated will depend on whether you are explicitly asked to express your observations in this way.

Problems to Avoid

Overgeneralization One of the goals of a case study is to lay a foundation for understanding broader trends and issues applied to similar circumstances. However, be careful when drawing conclusions from your case study. They must be evidence-based and grounded in the results of the study; otherwise, it is merely speculation. Looking at a prior example, it would be incorrect to state that a factor in improving girls access to education in Azerbaijan and the policy implications this may have for improving access in other Muslim nations is due to girls access to social media if there is no documentary evidence from your case study to indicate this. There may be anecdotal evidence that retention rates were better for girls who were engaged with social media, but this observation would only point to the need for further research and would not be a definitive finding if this was not a part of your original research agenda.

Failure to Document Limitations No case is going to reveal all that needs to be understood about a research problem. Therefore, just as you have to clearly state the limitations of a general research study , you must describe the specific limitations inherent in the subject of analysis. For example, the case of studying how women conceptualize the need for water conservation in a village in Uganda could have limited application in other cultural contexts or in areas where fresh water from rivers or lakes is plentiful and, therefore, conservation is understood more in terms of managing access rather than preserving access to a scarce resource.

Failure to Extrapolate All Possible Implications Just as you don't want to over-generalize from your case study findings, you also have to be thorough in the consideration of all possible outcomes or recommendations derived from your findings. If you do not, your reader may question the validity of your analysis, particularly if you failed to document an obvious outcome from your case study research. For example, in the case of studying the accident at the railroad crossing to evaluate where and what types of warning signals should be located, you failed to take into consideration speed limit signage as well as warning signals. When designing your case study, be sure you have thoroughly addressed all aspects of the problem and do not leave gaps in your analysis that leave the reader questioning the results.

Case Studies. Writing@CSU. Colorado State University; Gerring, John. Case Study Research: Principles and Practices . New York: Cambridge University Press, 2007; Merriam, Sharan B. Qualitative Research and Case Study Applications in Education . Rev. ed. San Francisco, CA: Jossey-Bass, 1998; Miller, Lisa L. “The Use of Case Studies in Law and Social Science Research.” Annual Review of Law and Social Science 14 (2018): TBD; Mills, Albert J., Gabrielle Durepos, and Eiden Wiebe, editors. Encyclopedia of Case Study Research . Thousand Oaks, CA: SAGE Publications, 2010; Putney, LeAnn Grogan. "Case Study." In Encyclopedia of Research Design , Neil J. Salkind, editor. (Thousand Oaks, CA: SAGE Publications, 2010), pp. 116-120; Simons, Helen. Case Study Research in Practice . London: SAGE Publications, 2009;  Kratochwill,  Thomas R. and Joel R. Levin, editors. Single-Case Research Design and Analysis: New Development for Psychology and Education .  Hilldsale, NJ: Lawrence Erlbaum Associates, 1992; Swanborn, Peter G. Case Study Research: What, Why and How? London : SAGE, 2010; Yin, Robert K. Case Study Research: Design and Methods . 6th edition. Los Angeles, CA, SAGE Publications, 2014; Walo, Maree, Adrian Bull, and Helen Breen. “Achieving Economic Benefits at Local Events: A Case Study of a Local Sports Event.” Festival Management and Event Tourism 4 (1996): 95-106.

Writing Tip

At Least Five Misconceptions about Case Study Research

Social science case studies are often perceived as limited in their ability to create new knowledge because they are not randomly selected and findings cannot be generalized to larger populations. Flyvbjerg examines five misunderstandings about case study research and systematically "corrects" each one. To quote, these are:

Misunderstanding 1 :  General, theoretical [context-independent] knowledge is more valuable than concrete, practical [context-dependent] knowledge. Misunderstanding 2 :  One cannot generalize on the basis of an individual case; therefore, the case study cannot contribute to scientific development. Misunderstanding 3 :  The case study is most useful for generating hypotheses; that is, in the first stage of a total research process, whereas other methods are more suitable for hypotheses testing and theory building. Misunderstanding 4 :  The case study contains a bias toward verification, that is, a tendency to confirm the researcher’s preconceived notions. Misunderstanding 5 :  It is often difficult to summarize and develop general propositions and theories on the basis of specific case studies [p. 221].

While writing your paper, think introspectively about how you addressed these misconceptions because to do so can help you strengthen the validity and reliability of your research by clarifying issues of case selection, the testing and challenging of existing assumptions, the interpretation of key findings, and the summation of case outcomes. Think of a case study research paper as a complete, in-depth narrative about the specific properties and key characteristics of your subject of analysis applied to the research problem.

Flyvbjerg, Bent. “Five Misunderstandings About Case-Study Research.” Qualitative Inquiry 12 (April 2006): 219-245.

• << Previous: Writing a Case Analysis Paper
• Next: Writing a Field Report >>
• Last Updated: Mar 6, 2024 1:00 PM
• URL: https://libguides.usc.edu/writingguide/assignments

• SUGGESTED TOPICS
• The Magazine
• Newsletters
• Managing Yourself
• Managing Teams
• Work-life Balance
• The Big Idea
• Data & Visuals
• Reading Lists
• Case Selections
• HBR Learning
• Topic Feeds
• Account Settings
• Email Preferences

What the Case Study Method Really Teaches

• Nitin Nohria

Seven meta-skills that stick even if the cases fade from memory.

It’s been 100 years since Harvard Business School began using the case study method. Beyond teaching specific subject matter, the case study method excels in instilling meta-skills in students. This article explains the importance of seven such skills: preparation, discernment, bias recognition, judgement, collaboration, curiosity, and self-confidence.

During my decade as dean of Harvard Business School, I spent hundreds of hours talking with our alumni. To enliven these conversations, I relied on a favorite question: “What was the most important thing you learned from your time in our MBA program?”

• Nitin Nohria is the George F. Baker Professor of Business Administration, Distinguished University Service Professor, and former dean of Harvard Business School.

Partner Center

• Harvard Business School →
• Academic Experience
• Faculty & Research
• The Field Method
• A Global Experience

The HBS Case Method

• Joint Degree Programs
• The Section Experience
• The HBS Case Method →

Take a Seat in the MBA Classroom

• Harvard Business School

How the HBS Case Method Works

How the Case Method Works

• Read and analyze the case. Each case is a 10-20 page document written from the viewpoint of a real person leading a real organization. In addition to background information on the situation, each case ends in a key decision to be made. Your job is to sift through the information, incomplete by design, and decide what you would do.
• Discuss the case. Each morning, you’ll bring your ideas to a small team of classmates from diverse professional backgrounds, your discussion group, to share your findings and listen to theirs. Together, you begin to see the case from different perspectives, better preparing you for class.
• Engage in class. Be prepared to change the way you think as you debate with classmates the best path forward for this organization. The highly engaged conversation is facilitated by the faculty member, but it’s driven by your classmates’ comments and experiences. HBS brings together amazingly talented people from diverse backgrounds and puts that experience front and center. Students do the majority of the talking (and lots of active listening), and your job is to better understand the decision at hand, what you would do in the case protagonist’s shoes, and why. You will not leave a class thinking about the case the same way you thought about it coming in! In addition to learning more about many businesses, in the case method you will develop communication, listening, analysis, and leadership skills. It is a truly dynamic and immersive learning environment.
• Reflect. The case method prepares you to be in leadership positions where you will face time-sensitive decisions with limited information. Reflecting on each class discussion will prepare you to face these situations in your future roles.

Student Perspectives

“I’ve been so touched by how dedicated other people have been to my learning and my success.”

Faculty Perspectives

“The world desperately needs better leadership. It’s actually one of the great gifts of teaching here, you can do something about it.”

Alumni Perspectives

“You walk into work every morning and it's like a fire hose of decisions that need to be made, often without enough information. Just like an HBS case.”

Celebrating the Inaugural HBS Case

“How do you go into an ambiguous situation and get to the bottom of it? That skill – the skill of figuring out a course of inquiry, to choose a course of action – that skill is as relevant today as it was in 1921.”

Top 40 Most Popular Case Studies of 2021

Two cases about Hertz claimed top spots in 2021's Top 40 Most Popular Case Studies

Two cases on the uses of debt and equity at Hertz claimed top spots in the CRDT’s (Case Research and Development Team) 2021 top 40 review of cases.

Hertz (A) took the top spot. The case details the financial structure of the rental car company through the end of 2019. Hertz (B), which ranked third in CRDT’s list, describes the company’s struggles during the early part of the COVID pandemic and its eventual need to enter Chapter 11 bankruptcy. 

The success of the Hertz cases was unprecedented for the top 40 list. Usually, cases take a number of years to gain popularity, but the Hertz cases claimed top spots in their first year of release. Hertz (A) also became the first ‘cooked’ case to top the annual review, as all of the other winners had been web-based ‘raw’ cases.

Besides introducing students to the complicated financing required to maintain an enormous fleet of cars, the Hertz cases also expanded the diversity of case protagonists. Kathyrn Marinello was the CEO of Hertz during this period and the CFO, Jamere Jackson is black.

Sandwiched between the two Hertz cases, Coffee 2016, a perennial best seller, finished second. “Glory, Glory, Man United!” a case about an English football team’s IPO made a surprise move to number four.  Cases on search fund boards, the future of malls,  Norway’s Sovereign Wealth fund, Prodigy Finance, the Mayo Clinic, and Cadbury rounded out the top ten.

Other year-end data for 2021 showed:

• Online “raw” case usage remained steady as compared to 2020 with over 35K users from 170 countries and all 50 U.S. states interacting with 196 cases.
• Fifty four percent of raw case users came from outside the U.S..
• The Yale School of Management (SOM) case study directory pages received over 160K page views from 177 countries with approximately a third originating in India followed by the U.S. and the Philippines.
• Twenty-six of the cases in the list are raw cases.
• A third of the cases feature a woman protagonist.
• Orders for Yale SOM case studies increased by almost 50% compared to 2020.
• The top 40 cases were supervised by 19 different Yale SOM faculty members, several supervising multiple cases.

CRDT compiled the Top 40 list by combining data from its case store, Google Analytics, and other measures of interest and adoption.

All of this year’s Top 40 cases are available for purchase from the Yale Management Media store .

And the Top 40 cases studies of 2021 are:

1.   Hertz Global Holdings (A): Uses of Debt and Equity

2.   Coffee 2016

3.   Hertz Global Holdings (B): Uses of Debt and Equity 2020

4.   Glory, Glory Man United!

5.   Search Fund Company Boards: How CEOs Can Build Boards to Help Them Thrive

6.   The Future of Malls: Was Decline Inevitable?

7.   Strategy for Norway's Pension Fund Global

8.   Prodigy Finance

9.   Design at Mayo

10. Cadbury

11. City Hospital Emergency Room

13. Volkswagen

14. Marina Bay Sands

15. Shake Shack IPO

16. Mastercard

17. Netflix

18. Ant Financial

19. AXA: Creating the New CR Metrics

20. IBM Corporate Service Corps

21. Business Leadership in South Africa's 1994 Reforms

22. Alternative Meat Industry

23. Children's Premier

24. Khalil Tawil and Umi (A)

25. Palm Oil 2016

26. Teach For All: Designing a Global Network

27. What's Next? Search Fund Entrepreneurs Reflect on Life After Exit

28. Searching for a Search Fund Structure: A Student Takes a Tour of Various Options

30. Project Sammaan

31. Commonfund ESG

32. Polaroid

33. Connecticut Green Bank 2018: After the Raid

34. FieldFresh Foods

35. The Alibaba Group

36. 360 State Street: Real Options

37. Herman Miller

38. AgBiome

39. Nathan Cummings Foundation

40. Toyota 2010

What is the Case Study Method?

Overview Dropdown up

Overview dropdown down, celebrating 100 years of the case method at hbs.

The 2021-2022 academic year marks the 100-year anniversary of the introduction of the case method at Harvard Business School. Today, the HBS case method is employed in the HBS MBA program, in Executive Education programs, and in dozens of other business schools around the world. As Dean Srikant Datar's says, the case method has withstood the test of time.

Case Discussion Preparation Details Expand All Collapse All

In self-reflection in self-reflection dropdown down, in a small group setting in a small group setting dropdown down, in the classroom in the classroom dropdown down, beyond the classroom beyond the classroom dropdown down, how the case method creates value dropdown up, how the case method creates value dropdown down, in self-reflection, in a small group setting, in the classroom, beyond the classroom.

How Cases Unfold In the Classroom

How cases unfold in the classroom dropdown up, how cases unfold in the classroom dropdown down, preparation guidelines expand all collapse all, read the professor's assignment or discussion questions read the professor's assignment or discussion questions dropdown down, read the first few paragraphs and then skim the case read the first few paragraphs and then skim the case dropdown down, reread the case, underline text, and make margin notes reread the case, underline text, and make margin notes dropdown down, note the key problems on a pad of paper and go through the case again note the key problems on a pad of paper and go through the case again dropdown down, how to prepare for case discussions dropdown up, how to prepare for case discussions dropdown down, read the professor's assignment or discussion questions, read the first few paragraphs and then skim the case, reread the case, underline text, and make margin notes, note the key problems on a pad of paper and go through the case again, case study best practices expand all collapse all, prepare prepare dropdown down, discuss discuss dropdown down, participate participate dropdown down, relate relate dropdown down, apply apply dropdown down, note note dropdown down, understand understand dropdown down, case study best practices dropdown up, case study best practices dropdown down, participate, what can i expect on the first day dropdown down.

Most programs begin with registration, followed by an opening session and a dinner. If your travel plans necessitate late arrival, please be sure to notify us so that alternate registration arrangements can be made for you. Please note the following about registration:

HBS campus programs – Registration takes place in the Chao Center.

India programs – Registration takes place outside the classroom.

Other off-campus programs – Registration takes place in the designated facility.

What happens in class if nobody talks? Dropdown down

Professors are here to push everyone to learn, but not to embarrass anyone. If the class is quiet, they'll often ask a participant with experience in the industry in which the case is set to speak first. This is done well in advance so that person can come to class prepared to share. Trust the process. The more open you are, the more willing you’ll be to engage, and the more alive the classroom will become.

Does everyone take part in "role-playing"? Dropdown down

Professors often encourage participants to take opposing sides and then debate the issues, often taking the perspective of the case protagonists or key decision makers in the case.

View Frequently Asked Questions

Subscribe to Our Emails

• Leadership Team
• Our Approach
• PMO Services
• Program & Project Managers
• PM Training & Development
• Case Studies
• Our Culture
• Employee Benefits
• Job Opportunities
• Benefits Realization Management Diagnostic
• Project Management Maturity Guide
• Guide to Agile Project Management
• PM Maturity Assessment
• Project Management as a Service (PMaaS) and Why It’s the Future

About PM Solutions

PM Solutions is a project management consulting firm that helps PMO, project, and business leaders apply project and portfolio management practices that drive performance and operational efficiency.

• Co-Founder & Co-CEO J. Kent Crawford
• Co-Founder & Co-CEO Deborah Bigelow Crawford
• President, PM Solutions & PM College Bruce Miller
• Vice President, Client Success, Eric Foss
• Managing Director, HR & Administration, Karen Alfonsi
• Director, Marketing and Communications, Carrie Capili

With our approach , companies can expect high-value, high-impact solutions, and measurable, sustainable results.

• PMO Deployment, Operation, and Enhancement
• Project Management as a Service (PMaaS)
• Project Review & Recovery
• Project Portfolio Management (PPM)
• Project Management Maturity Advancement
• Organizational Change Management
• Project Management Methodology Implementation
• Demand Management
• Project Management Mentors
• Resource Management
• Vendor Management

Project & Program Managers

We can provide you with highly experienced program and project managers ; experts to help guide, lead, and support high-visibility initiatives.

PM Training & Development

PM College® provides corporate project management training and competency programs for clients around the world.

By Project Initiatives

• Cost Reduction Initiatives (1)
• Data Center Consolidation (1)
• High-risk Capital Initiatives (1)
• Infrastructure Program Management (0)
• Manufacturing Facility Operations (1)
• Mentoring (11)
• Methodology (4)
• New Product Development (1)
• Organizational Change (6)
• PMO Assessment (4)
• PMO Deployment (4)
• Process Improvement (7)
• Program & Portfolio Management (10)
• Project Audits (1)
• Project Management Training (7)
• Regulatory Compliance (1)
• Resource Management (1)
• Strategy Execution (1)
• Systems Integration Deployment (0)
• Troubled Project Recovery (3)
• Vendor Management (2)

By Industry

• Automotive (1)
• Energy & Utilities (7)
• Financial Services (2)
• Human Resources (1)
• Information Technology (5)
• Insurance (5)
• Manufacturing (6)
• Pharma/Biotech (2)
• Professional Services (2)
• Research and Development (1)
• Retail & Merchandise (1)
• Security (1)
• Benefits Realization (4)
• Change Management (6)
• IT Project Management (4)
• Outsourcing Project Management (4)
• Performance & Value Measurement (12)
• Project Management Maturity (23)
• Project Management Methodology (12)
• Project Management Office (58)
• Project Management Training (31)
• Project Management Trends (50)
• Project Manager Competency (18)
• Project Portfolio Management (11)
• Project Recovery (9)
• Resource Management (5)
• Strategy & Governance (14)
• Articles (46)
• Brochures (3)
• eNewsletters (19)
• Research (44)
• Webinars (24)
• White Papers (35)
• contact us get in touch call: 800.983.0388

Home » Case Studies » Program & Portfolio Management

More program & portfolio management case studies, ontario power generation achieves project excellence through the establishment of an enterprise project management office.

"We have really changed public perception, and as a result, increased our credibility in OPG’s ability to execute projects. “- Mike Martelli, Chief Projects Officer, OPG Read More »

• Energy & Utilities
• PMO Assessment
• Process Improvement
• Program & Portfolio Management

‘Never Waste a Good Crisis’: Kettering Health Network Doubles Down on Project Management Improvement

Agile transformation of the PMO and transforming how Agile Scrum was used helped the PMO complete 103 projects, including a new hospital and a new cancer treatment center. It grew to a Level 3 PMO from a 1.8 PMO within a 12-month period. As the PMO transitions to an EPMO, this mission-critical function will have a broader strategic footprint, doubling its influence and impact across the network’s portfolio of projects over the next couple of years. Read More »

• Pharma/Biotech
• Organizational Change

Transformative Leadership Sets Financial Services Enterprise PMO on Fast Track to Strategic Value

For this privately held financial services company, a compelling journey of business transformation started with a good read. Read More »

• Financial Services
• PMO Deployment
• Project Management Training
• Strategy Execution

Accident Fund’s Award-Winning PMO Puts Processes Before Tools in Implementing Project Portfolio Management

PM Solutions used its PPM Maturity and Project Management Maturity Models to develop an improvement roadmap that included a number of specific recommendations from a project management governance perspective. Established PPM processes that helped eliminate approximately 100 non-value-adding projects, led to the successful implementation of a PPM tool, and won the company national acclaim – all in under three years. Read More »

Claims Management Company Successfully Completes Massive M&A Initiative Using Best-Practice Project and Program Management

The company grew dramatically by successfully integrating more than 10,000 new customers, opening two additional call centers, and doubling the number of claims serviced — while meeting program deadlines, contractual obligations, and increasing revenues by over 30%. Read More »

• Information Technology
• Methodology
• Troubled Project Recovery

Fast-Track PMO Implementation Rescues Troubled Projects and Improves Customer Satisfaction

For one of the largest privately-held staffing companies in the United States: PM Solutions implemented a fully-functional PMO including Project Portfolio Management tools after an unsuccessful first attempt at implementing a PMO on behalf of the Company. Value Delivered: The resulting visibility into projects allowed the client to recover 100% of the troubled projects in a portfolio valued at approximately $13M, while significantly improving project manager morale as well as project performance and customer satisfaction. Read More »

• Professional Services

Improvements in Project and Portfolio Management Yield Immediate Returns

Dramatic increases in project execution capabilities and productivity resulted in $900,000 of operational cost avoidance during the current budget year. Read More »

• Project Audits

Project Management Maturity Practices Yield Improved Operational Efficiencies

For a multi-billion dollar strategy and technology consulting firm: PM Solutions performed a project management maturity assessment, delivered an implementation plan to improve maturity, provided portfolio management consulting and project management training, and performed a re-assessment to gauge improvement over a three-year period. Value Delivered: Overall process maturity improved by 40%. Read More »

New Manufacturing Facility Goes Online in Record Time

PM Solutions is providing project and program management to bring a state-of-the-art, $60 million dollar manufacturing facility online and fully operational in record time. Value Delivered: The facility was operational in just 12 months, allowing the client to realize 33% of their project return seven months ahead of schedule. Read More »

• Manufacturing
• Manufacturing Facility Operations

Program Meets Regulatory Requirements Ahead of Schedule and Under Budget

PM Solutions provided program management of a product reformulation initiative that affected $260 million (Net Sales Value) worth of products driven by regulatory compliance. Value Delivered: The program was completed six months ahead of schedule and $2 million under budget. Read More »

• Regulatory Compliance
• eNewsletters
• White Papers
• PMO of the Year Award

Connect with Us

The project management experts®.

PM Solutions is a project management services firm helping organizations apply project management and PMO practices to improve business performance.

Toll-free (US): 800.983.0388 International: +1.484.450.0100 [email protected]

285 Wilmington-West Chester Pike Chadds Ford, PA 19317 USA

© 2012-2024 Project Management Solutions, Inc. All Rights Reserved. Privacy & Terms

• --> Login or Sign Up

Shop by Author

• Sabrineh Ardalan
• Robert Bordone
• Robert Clark
• John Coates
• Susan Crawford
• Alonzo Emery
• Heidi Gardner
• Philip B. Heymann
• Howell E. Jackson
• Wendy Jacobs
• Adriaan Lanni
• Jeremy McClane
• Naz Modirzadeh
• Catherine Mondell
• Ashish Nanda
• Charles R. Nesson
• John Palfrey
• Bruce Patton
• Todd D. Rakoff
• Lisa Rohrer
• Jeswald W. Salacuse
• James Sebenius
• Joseph William Singer
• Holger Spamann
• Carol Steiker
• Guhan Subramanian
• Lawrence Susskind
• David B. Wilkins
• Jonathan Zittrain

Shop by Brand

Howell Jackson

• Ashish Nanda and Nicholas Semi Haas
• Chad M. Carr
• John Coates, Clayton Rose, and David Lane
• Ashish Nanda and Lauren Prusiner
• Ashish Nanda and Lisa Rohrer
• Ashish Nanda and Monet Brewerton
• View all Brands
• $0.00 - $10.00
• $10.00 - $19.00
• $19.00 - $29.00
• $29.00 - $38.00
• $38.00 - $48.00

Case Studies Program

The Case Studies Program supports a wide range of case development projects throughout Harvard Law School. Faculty at HLS are in the process of creating innovative, experiential materials for the legal classroom, and the Case Studies Program works with them to conceive, develop, edit, and publish their classroom materials.

• Published Old-New
• Published New-Old

Investor Access to Private Investment

Hannah Valentine under the supervision of Howell Jackson

First National Bank of Ames Corporation

Project Merchandise: An Introduction to Private Equity

Holger Spamann and Johnathan Robertson

Independent Investigations and Reports: Legal and Ethical Constraints

CLOUD Act Enforcement

Ryan Chan-Wei and Sebastian Steuer under the supervision of Howell E. Jackson

Regulating Consumer Permissioned Access to Financial Data

Nafisa Abubakar Adama and Connor Tweardy under the supervision of Howell E. Jackson

Mobile Payments for the Developing World

Jonathan Greenacre with Howell E. Jackson

Machine Learning in the Underwriting of Consumer Loans

Yinan Liu and Talia Gillis under the supervision of Howell E. Jackson

Regulating Crypto Assets: Securities and Regulations

Amy Aixi Zhang under Howell Jackson

Market Manipulation: Definitional Approaches

Christina Drakeford under Howell Jackson

Employee Benefits — Emergency Savings Account

Adam Spiegel under the supervision of Howell Jackson

America’s Most Wanted Informant: The FBI and the Case of Whitey Bulger Screen Reader Version

Georgia Stasinopoulos, under supervision of Philip Heymann

Noorain Khan and Disability Inclusion at the Ford Foundation

Laura Winig and Susan Crawford

BYOB in Boston

Susan Crawford and Brittany Deitch

Robo Advising

Ryan Chan-Wei HLS LLM '19 under the supervision of Howell Jackson

Fintech Charter

Margaret Tahyar, Madison Roberts, and Carol Rodrigues of Davis Polk & Wardwell, with the assistance of Professor Howell E. Jackson

Strategic Options and Legal Risks for Elite ReFi, Inc.

Margaret Tahyar, Jai Massari, Madison Roberts, and Adam Fovent of Davis Polk & Wardwell, with the assistance of Professor Howell E. Jackson

Digital Currencies

Margaret E. Tahyar, Trevor I. Kiviat, Madison J. Roberts, and Suiwen Liang of Davis, Polk and Wardwell with assistance by Howell Jackson

OUR Walmart

Sharon Block, Lisa Brem, and Brittany Deitch

Worker Centers

Friendly Savings Bank

Holger Spamann and Stanley Ragalevsky

Exercise on Alternative Strategies for Depository Institutions

Sanctuary Cities

Brittany Deitch and Lisa Brem, under the supervision of Sabrineh Ardalan

Wells Fargo Corporate Governance

Ed Stein under the supervision of Howell Jackson

Anti-Money Laundering and Blockchain Technology

Dylan M. Aluise under the supervision of Howell Jackson

The Future of Affiliate Transaction Restrictions for Banks and the Federal Reserve’s Emergency Intervention Authority

Margaret Tahyar and Howell Jackson

Closed-End Fund Regulation

Daniel Wertman under the Supervision of Howell Jackson

Unidentified Financial Institutions

Howell Jackson and Peter Tufano with assistance from Andrea Ryan

Asset Securitization, Marketplace Lending, and the Future After the Madden Decision

Consumer Financial Protection Bureau: Responding to the PHH Litigation

Ryan M. Johansen under the supervision of Howell Jackson

Wal-Mart and Banking

Joshua Cutler under the supervision of Howell Jackson

Lending Club: 2008

Anooshree Sinha and Corrine Snow under the supervision of Howell Jackson

Ames’ Auto Insurance Regulations — Racial Disparities in Insurance Premiums

Federal Preemption of State Consumer Protection Laws: The Office of the Comptroller of the Currency's Dodd-Frank Act Implementation Rules

Melanie Berdecia and Dylan Aluise under supervision of Howell Jackson

The OCC’s FinTech Charter: Testing the Scope of the OCC’s Chartering Powers

Sexual Assault Policy in Higher Education: An Introduction

Elizabeth Moroney, under supervision of Philip B. Heymann

A Rolling Stone Controversy: Sexual Assault Policy at the University of Virginia (B) and (C)

A Rolling Stone Controversy: Sexual Assault Policy at the University of Virginia (A)

The Organized Crime Cases: Moves and Countermoves

Philip Heymann

The Colombia Connection: An International Solution for an International Problem

Sean Driscoll, under supervision of Philip Heymann

Albert Gonzalez: Get Rich or Die Tryin’

Sharon Kim, under supervision of Philip Heymann

Liberty Reserve: The Face of Cyber-Laundering

Nicholas Maietta, under supervision of Philip Heymann

America’s Most Wanted Informant: The FBI and the Case of Whitey Bulger

Turf Wars and the Ciudad Juárez Cartel

Amanda Liverzani, under supervision of Philip Heymann

The Growth of Organized Crime in the United States

Alisha Crovetto, Ari Ruben, and Elizabeth Moroney, under supervision of Philip Heymann

Los Zetas and a Strategy Built on Fear

Sophie Elsner, under supervision of Philip Heymann

Sinaloa Cartel, A Business Strategy

Amanda Liverzani and Michael Soyfer, under supervision of Philip Heymann

What’s Fair about Fair Use? The Battle over E-Reserves at GSU (B)

Elizabeth Moroney, under the supervision of Kyle Courtney and William Fisher

Prosecutorial Discretion in Charging and Plea Bargaining: The Aaron Swartz Case (B)

Elizabeth Moroney, under the supervision of Adriaan Lanni and Carol Steiker

What’s Fair about Fair Use? The Battle over E-Reserves at GSU (A)

The Battle for Unocal

Holger Spamann, Amanda Ravich, and Lisa Brem

Prosecutorial Discretion in Charging and Plea Bargaining: The Aaron Swartz Case (A)

Elizabeth Moroney, under supervision of Adriaan Lanni and Carol Steiker

Reputation, Credibility, and the Goldstone Report (B)

Philip Heymann, Taylor Lane, and Lisa Brem

Reputation, Credibility, and the Goldstone Report (A)

Margaret Hamburg and the FDA

Philip Heymann and Lisa Brem

Cyrus Vance and Dominique Strauss-Kahn: Dilemmas in a High-Profile Prosecution (B) and (C)

Philip Heymann, Lisa Brem, Emily Balter, and Michael Noveck

Cyrus Vance and Dominique Strauss-Kahn: Dilemmas in a High-Profile Prosecution (A)

• Login Start free trial

Starbucks Loyalty Program Case Study

Starbucks, renowned as one of the foremost coffee brands globally, owes its success not just to its exceptional range of products. A significant contributor to its widespread acclaim is its highly popular loyalty program , which has been an inspiration for many U.S. restaurant chains and retail stores. Launched on December 26th, 2009, the Starbucks rewards program has amassed over 30 million active members, contributing to nearly 60% of the brand's total revenue. The program effectiveness was strikingly evident in 2019, with Starbucks reporting a substantial 7% increase in sales primarily attributed to the efficacy of its rewards program.

Program type 

Points system.

• Points for rewards
• Coalition Loyalty

Starbucks Rewards leverages a points-based system ingeniously designed to enhance customer engagement and loyalty. Members accumulate 'Stars' with each purchase made via their registered Starbucks Card or through the Starbucks mobile app.  The program is structured to encourage varied forms of engagement, enhancing the overall customer experience and brand loyalty.

• Earn 1 Star per $1 spent when you pay in cash
• Earn 2 Stars per $1 spent when you preload your digital Starbucks Card
• Earn additional bonus Stars through special promotions
• Earn double Stars on Delta Airlines travel days
• Earn double Stars on promotional days
• Points expire after 6 months

Starbucks Rewards allows its members to redeem their accumulated Stars at various levels, each unlocking distinct rewards. This structure caters to diverse preferences, allowing customers to either redeem their Stars for immediate benefits or save them for more substantial rewards.

• 25 Stars: Redeem for a free drink customization, such as extra flavor, espresso shot, or alternative milk.
• 50 Stars: Get a complimentary hot coffee, tea, or bakery item like a bagel or croissant.
• 150 Stars: Choose a free handcrafted drink, hot breakfast, or a parfait.
• 200 Stars: Earn a complimentary lunch item, like a salad, sandwich, or protein box.
• 400 Stars: Exchange for packaged coffee beans or select Starbucks merchandise.

Coalition loyalty

Starbucks and Delta Airlines have embarked on a partnership, merging the world of coffee with the skies. Linking your Delta SkyMiles® and Starbucks® Rewards accounts opens up a new realm of earning opportunities and exclusive benefits.

• Travel-day perks: Double Stars on Delta travel days
• Everyday earn: 1 mile per $1* spent at Starbucks
• Exclusive offers: More opportunities to earn Stars and miles

Interesting takeaways

Multi-channel communication.

Starbucks leverages multiple channels to communicate about its rewards program, ensuring widespread awareness and engagement. The use of a dedicated landing page, mobile app, points balance reminder emails , company blogs, and social media platforms creates an ecosystem where information about the program is readily accessible and consistently updated. The clarity and frequency of updates, including press releases for significant changes, demonstrate Starbucks’ commitment to transparency. This approach not only keeps existing members informed but also attracts new participants by clearly outlining the program’s benefits. Good communication is likely a significant factor in the program's high membership numbers.

Program simplicity and incentive structure

The simplicity of the points system (earn points and redeem for rewards) is user-friendly and easy for customers to grasp. This clarity enhances the appeal of the program and encourages participation. The structured nature of rewards incentivizes customers to accumulate more points to access greater rewards. This gamification element adds an extra layer of engagement, as customers are motivated to reach the next level. As customers get closer to a significant reward, their motivation to attain it increases. This psychological aspect of goal setting and achievement is a powerful tool in enhancing customer loyalty and encouraging more frequent purchases.

Incentivizing preloaded payments

Starbucks ingeniously incentivizes customers to preload their Starbucks account by offering double points for purchases made using a preloaded Starbucks Card. By preloading their accounts, customers commit themselves financially to future purchases at Starbucks, which effectively guarantees a degree of customer loyalty and repeat business. This strategy turns occasional visitors into regular customers, as the preloaded funds will likely be used exclusively at Starbucks locations. From a financial perspective, this strategy is beneficial for Starbucks. When customers preload their accounts, the brand receives cash in advance, improving its cash flow. This tactic also promotes increased engagement with the Starbucks mobile app, where preloading and tracking of rewards can be easily managed. 

Coalition program with Delta Airlines

The partnership with Delta SkyMiles is a strategic move that broadens the program’s appeal. By collaborating with an airline, Starbucks extends its reach beyond the typical coffee shop customer. This alliance allows both companies to tap into each other’s customer bases, potentially attracting new customers who are loyal to one brand to become interested in the other. It’s a creative way to expand market reach without direct competition. For customers who already patronize both brands, this coalition loyalty program reinforces their loyalty and increases their engagement with both Starbucks and Delta. It creates a more integrated customer experience and can deepen the sense of brand loyalty.

Areas of improvement

The removal of the tiers system.

The previous iteration of the program had a 'Gold' tier, which was a smart way to acknowledge and reward the most loyal customers. By offering them the ability to earn points faster and providing exclusive benefits, Starbucks recognized and cherished its most valuable customers. This tier system not only incentivized increased spending but also gave a sense of exclusivity and status. While simplifying the program might make it more accessible for new customers, it risks diminishing the perceived value for power shoppers. The elimination of the tier system could lead to a decrease of engagement from these highly valuable customers.

Lack of personalized engagement or rewards

In the current market, personalization is key to customer engagement. Starbucks’ generic approach to rewards and communication, without considering individual customer preferences and behaviors, is a missed opportunity. Personalizing rewards and communications could significantly enhance customer loyalty and encourage repeat business. Leveraging customer data to offer personalized rewards and product recommendations could not only improve customer satisfaction but also drive product discovery and sales.

Limited post-purchase engagement

Post-purchase engagement is crucial in maintaining a connection with the customer even after they leave the store. This ongoing engagement is essential for converting first-time buyers into regular customers. While Starbucks has started initiatives like Starbucks Odyssey , which offers additional ways to earn rewards through digital journeys, its integration with the main rewards program appears to be lacking. A more seamless integration could enhance the overall experience and effectiveness of the loyalty program. Recognizing that some of these initiatives are in their early stages, there is potential for growth and improvement.

Learn from our library of loyalty program case studies

10 Ways to Increase Repeat Purchases on your Online Store

Repeat customers are the backbone of almost every successful business. Here are ten ways you can increase repeated purchases made in your store.

Sephora Beauty Insider Loyalty Program Review

While Sephora’s Beauty Insider has maintained its position of one of the most successful loyalty programs, it has gone through several phases to ensure that its customer base is glued to the benefits and experiences.

Ready to grow your business?

This site uses cookies to store information on your computer. Some are essential to make our site work; others help us improve the user experience. By using the site, you consent to the placement of these cookies.

An official website of the United States government, Department of Justice.

Here's how you know

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock A locked padlock ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Cross-Site Analysis and Case Study of STOP Program Grantee Perspectives on Violence Prevention and Mental Health Training Program Implementation

This paper explores STOP Program grantee perspectives on violence prevention and mental health training program implementation.

In this study, researchers examined factors that influence the implementation of violence prevention and mental health training programs in schools, with a particular focus on implementation readiness and school mental health capacity. Several promising results were found that may contribute to ongoing efforts to improve school safety. In response to the Students, Teachers, and Officers Preventing (STOP) School Violence Act of 2018 (H.R. 4909), 128 grantees across the U.S. were awarded funding through the Bureau of Justice Assistance (BJA) in 2018 and 2019 to improve school safety by implementing programs in the Violence Prevention and Mental Health Training category. The major goals of this study were to 1) understand the challenges and facilitators of implementing violence prevention and mental health training programs through a broad cross-site analysis, 2) assess contextual factors influencing implementation, as well as regional and population variances through targeted, comprehensive case studies, and 3) provide evidence to inform program implementation in violence prevention and mental health programs in schools to improve program outcomes and sustainability. Understanding the environment of implementation, grantees’ capacity to carry out planned activities, and the perspectives of implementation team members are critical components to learning what factors support and inhibit implementation and ultimately, the extent to which programming will be replicable and scalable as federal funding continues to support mental health and violence prevention initiatives. The study was conducted at two levels: 1) a cross-site analysis of grantees who have been awarded funding in the Violence Prevention and Mental Health Training category over the two award years (2018 and 2019), and 2) a case study analysis of six grantee sites.

Additional Details

Related topics, similar publications.

• Adolescent Contact, Lasting Impact? Lessons Learned From Two Longitudinal Studies Spanning 20 Years of Developmental Science Research With Justice-System-Involved Youths
• Urban Adolescents' Exposure to Violence and Racial Discrimination: Gender Differences in Coping and Mental Health
• Common Problems, Common Solutions: Looking Across Sectors at Strategies for Supporting Rural Youth and Families Tool-Kit

Port Security and Emergency Response Using Autonomous Systems: Virginia Pilot Program

The Port Security and Emergency Response (PS&ER) Using Autonomous Systems (UxS) project focused on applying innovative UxS solutions to enhance the safety, security, and operational effectiveness of the Port of Virginia. This report documents a two-year collaborative effort to advance the use of autonomous vehicles from the conceptual stage to real-world demonstrations. The project began with a series of workshops to first identify and prioritize public safety and emergency response challenges and then generate requirements for UxS technologies to address these challenges. Next, the project funded five cross-domain solutions (air, ground, and maritime) that demonstrated UxS capabilities deployed to detect and observe public safety or emergency response events that affect the Port of Virginia. Based on the outcomes of the project, we explore a holistic integrated solution for PS&ER in the context of port operations and scenarios that enhances the safety, security, and overall effectiveness of the port.

Approved for public release. Unlimited distribution.

• Document Number: IPD-2024-U-038017
• Publication Date: 4/24/2024

• Open access
• Published: 22 April 2024

Artificial intelligence and medical education: application in classroom instruction and student assessment using a pharmacology & therapeutics case study

• Kannan Sridharan 1 &
• Reginald P. Sequeira 1  

BMC Medical Education volume  24 , Article number:  431 ( 2024 ) Cite this article

354 Accesses

1 Altmetric

Metrics details

Artificial intelligence (AI) tools are designed to create or generate content from their trained parameters using an online conversational interface. AI has opened new avenues in redefining the role boundaries of teachers and learners and has the potential to impact the teaching-learning process.

In this descriptive proof-of- concept cross-sectional study we have explored the application of three generative AI tools on drug treatment of hypertension theme to generate: (1) specific learning outcomes (SLOs); (2) test items (MCQs- A type and case cluster; SAQs; OSPE); (3) test standard-setting parameters for medical students.

Analysis of AI-generated output showed profound homology but divergence in quality and responsiveness to refining search queries. The SLOs identified key domains of antihypertensive pharmacology and therapeutics relevant to stages of the medical program, stated with appropriate action verbs as per Bloom’s taxonomy. Test items often had clinical vignettes aligned with the key domain stated in search queries. Some test items related to A-type MCQs had construction defects, multiple correct answers, and dubious appropriateness to the learner’s stage. ChatGPT generated explanations for test items, this enhancing usefulness to support self-study by learners. Integrated case-cluster items had focused clinical case description vignettes, integration across disciplines, and targeted higher levels of competencies. The response of AI tools on standard-setting varied. Individual questions for each SAQ clinical scenario were mostly open-ended. The AI-generated OSPE test items were appropriate for the learner’s stage and identified relevant pharmacotherapeutic issues. The model answers supplied for both SAQs and OSPEs can aid course instructors in planning classroom lessons, identifying suitable instructional methods, establishing rubrics for grading, and for learners as a study guide. Key lessons learnt for improving AI-generated test item quality are outlined.

Conclusions

AI tools are useful adjuncts to plan instructional methods, identify themes for test blueprinting, generate test items, and guide test standard-setting appropriate to learners’ stage in the medical program. However, experts need to review the content validity of AI-generated output. We expect AIs to influence the medical education landscape to empower learners, and to align competencies with curriculum implementation. AI literacy is an essential competency for health professionals.

Peer Review reports

Artificial intelligence (AI) has great potential to revolutionize the field of medical education from curricular conception to assessment [ 1 ]. AIs used in medical education are mostly generative AI large language models that were developed and validated based on billions to trillions of parameters [ 2 ]. AIs hold promise in the incorporation of history-taking, assessment, diagnosis, and management of various disorders [ 3 ]. While applications of AIs in undergraduate medical training are being explored, huge ethical challenges remain in terms of data collection, maintaining anonymity, consent, and ownership of the provided data [ 4 ]. AIs hold a promising role amongst learners because they can deliver a personalized learning experience by tracking their progress and providing real-time feedback, thereby enhancing their understanding in the areas they are finding difficult [ 5 ]. Consequently, a recent survey has shown that medical students have expressed their interest in acquiring competencies related to the use of AIs in healthcare during their undergraduate medical training [ 6 ].

Pharmacology and Therapeutics (P & T) is a core discipline embedded in the undergraduate medical curriculum, mostly in the pre-clerkship phase. However, the application of therapeutic principles forms one of the key learning objectives during the clerkship phase of the undergraduate medical career. Student assessment in pharmacology & therapeutics (P&T) is with test items such as multiple-choice questions (MCQs), integrated case cluster questions, short answer questions (SAQs), and objective structured practical examination (OSPE) in the undergraduate medical curriculum. It has been argued that AIs possess the ability to communicate an idea more creatively than humans [ 7 ]. It is imperative that with access to billions of trillions of datasets the AI platforms hold promise in playing a crucial role in the conception of various test items related to any of the disciplines in the undergraduate medical curriculum. Additionally, AIs provide an optimized curriculum for a program/course/topic addressing multidimensional problems [ 8 ], although robust evidence for this claim is lacking.

The existing literature has evaluated the knowledge, attitude, and perceptions of adopting AI in medical education. Integration of AIs in medical education is the need of the hour in all health professional education. However, the academic medical fraternity facing challenges in the incorporation of AIs in the medical curriculum due to factors such as inadequate grounding in data analytics, lack of high-quality firm evidence favoring the utility of AIs in medical education, and lack of funding [ 9 ]. Open-access AI platforms are available free to users without any restrictions. Hence, as a proof-of-concept, we chose to explore the utility of three AI platforms to identify specific learning objectives (SLOs) related to pharmacology discipline in the management of hypertension for medical students at different stages of their medical training.

Study design and ethics

The present study is observational, cross-sectional in design, conducted in the Department of Pharmacology & Therapeutics, College of Medicine and Medical Sciences, Arabian Gulf University, Kingdom of Bahrain, between April and August 2023. Ethical Committee approval was not sought given the nature of this study that neither had any interaction with humans, nor collection of any personal data was involved.

Study procedure

We conducted the present study in May-June 2023 with the Poe© chatbot interface created by Quora© that provides access to the following three AI platforms:

Sage Poe [ 10 ]: A generative AI search engine developed by Anthropic © that conceives a response based on the written input provided. Quora has renamed Sage Poe as Assistant © from July 2023 onwards.

Claude-Instant [ 11 ]: A retrieval-based AI search engine developed by Anthropic © that collates a response based on pre-written responses amongst the existing databases.

ChatGPT version 3.5 [ 12 ]: A generative architecture-based AI search engine developed by OpenAI © trained on large and diverse datasets.

We queried the chatbots to generate SLOs, A-type MCQs, integrated case cluster MCQs, integrated SAQs, and OSPE test items in the domain of systemic hypertension related to the P&T discipline. Separate prompts were used to generate outputs for pre-clerkship (preclinical) phase students, and at the time of graduation (before starting residency programs). Additionally, we have also evaluated the ability of these AI platforms to estimate the proportion of students correctly answering these test items. We used the following queries for each of these objectives:

Specific learning objectives

Can you generate specific learning objectives in the pharmacology discipline relevant to undergraduate medical students during their pre-clerkship phase related to anti-hypertensive drugs?

Can you generate specific learning objectives in the pharmacology discipline relevant to undergraduate medical students at the time of graduation related to anti-hypertensive drugs?

A-type MCQs

In the initial query used for A-type of item, we specified the domains (such as the mechanism of action, pharmacokinetics, adverse reactions, and indications) so that a sample of test items generated without any theme-related clutter, shown below:

Write 20 single best answer MCQs with 5 choices related to anti-hypertensive drugs for undergraduate medical students during the pre-clerkship phase of which 5 MCQs should be related to mechanism of action, 5 MCQs related to pharmacokinetics, 5 MCQs related to adverse reactions, and 5 MCQs should be related to indications.

The MCQs generated with the above search query were not based on clinical vignettes. We queried again to generate MCQs using clinical vignettes specifically because most medical schools have adopted problem-based learning (PBL) in their medical curriculum.

Write 20 single best answer MCQs with 5 choices related to anti-hypertensive drugs for undergraduate medical students during the pre-clerkship phase using a clinical vignette for each MCQ of which 5 MCQs should be related to the mechanism of action, 5 MCQs related to pharmacokinetics, 5 MCQs related to adverse reactions, and 5 MCQs should be related to indications.

We attempted to explore whether AI platforms can provide useful guidance on standard-setting. Hence, we used the following search query.

Can you do a simulation with 100 undergraduate medical students to take the above questions and let me know what percentage of students got each MCQ correct?

Integrated case cluster MCQs

Write 20 integrated case cluster MCQs with 2 questions in each cluster with 5 choices for undergraduate medical students during the pre-clerkship phase integrating pharmacology and physiology related to systemic hypertension with a case vignette.

Write 20 integrated case cluster MCQs with 2 questions in each cluster with 5 choices for undergraduate medical students during the pre-clerkship phase integrating pharmacology and physiology related to systemic hypertension with a case vignette. Please do not include ‘none of the above’ as the choice. (This modified search query was used because test items with ‘None of the above’ option were generated with the previous search query).

Write 20 integrated case cluster MCQs with 2 questions in each cluster with 5 choices for undergraduate medical students at the time of graduation integrating pharmacology and physiology related to systemic hypertension with a case vignette.

Integrated short answer questions

Write a short answer question scenario with difficult questions based on the theme of a newly diagnosed hypertensive patient for undergraduate medical students with the main objectives related to the physiology of blood pressure regulation, risk factors for systemic hypertension, pathophysiology of systemic hypertension, pathological changes in the systemic blood vessels in hypertension, pharmacological management, and non-pharmacological treatment of systemic hypertension.

Write a short answer question scenario with moderately difficult questions based on the theme of a newly diagnosed hypertensive patient for undergraduate medical students with the main objectives related to the physiology of blood pressure regulation, risk factors for systemic hypertension, pathophysiology of systemic hypertension, pathological changes in the systemic blood vessels in hypertension, pharmacological management, and non-pharmacological treatment of systemic hypertension.

Write a short answer question scenario with questions based on the theme of a newly diagnosed hypertensive patient for undergraduate medical students at the time of graduation with the main objectives related to the physiology of blood pressure regulation, risk factors for systemic hypertension, pathophysiology of systemic hypertension, pathological changes in the systemic blood vessels in hypertension, pharmacological management, and non-pharmacological treatment of systemic hypertension.

Can you generate 5 OSPE pharmacology and therapeutics prescription writing exercises for the assessment of undergraduate medical students at the time of graduation related to anti-hypertensive drugs?

Can you generate 5 OSPE pharmacology and therapeutics prescription writing exercises containing appropriate instructions for the patients for the assessment of undergraduate medical students during their pre-clerkship phase related to anti-hypertensive drugs?

Can you generate 5 OSPE pharmacology and therapeutics prescription writing exercises containing appropriate instructions for the patients for the assessment of undergraduate medical students at the time of graduation related to anti-hypertensive drugs?

Both authors independently evaluated the AI-generated outputs, and a consensus was reached. We cross-checked the veracity of answers suggested by AIs as per the Joint National Commission Guidelines (JNC-8) and Goodman and Gilman’s The Pharmacological Basis of Therapeutics (2023), a reference textbook [ 13 , 14 ]. Errors in the A-type MCQs were categorized as item construction defects, multiple correct answers, and uncertain appropriateness to the learner’s level. Test items in the integrated case cluster MCQs, SAQs and OSPEs were evaluated with the Preliminary Conceptual Framework for Establishing Content Validity of AI-Generated Test Items based on the following domains: technical accuracy, comprehensiveness, education level, and lack of construction defects (Table  1 ). The responses were categorized as complete and deficient for each domain.

The pre-clerkship phase SLOs identified by Sage Poe, Claude-Instant, and ChatGPT are listed in the electronic supplementary materials 1 – 3 , respectively. In general, a broad homology in SLOs generated by the three AI platforms was observed. All AI platforms identified appropriate action verbs as per Bloom’s taxonomy to state the SLO; action verbs such as describe, explain, recognize, discuss, identify, recommend, and interpret are used to state the learning outcome. The specific, measurable, achievable, relevant, time-bound (SMART) SLOs generated by each AI platform slightly varied. All key domains of antihypertensive pharmacology to be achieved during the pre-clerkship (pre-clinical) years were relevant for graduating doctors. The SLOs addressed current JNC Treatment Guidelines recommended classes of antihypertensive drugs, the mechanism of action, pharmacokinetics, adverse effects, indications/contraindications, dosage adjustments, monitoring therapy, and principles of monotherapy and combination therapy.

The SLOs to be achieved by undergraduate medical students at the time of graduation identified by Sage Poe, Claude-Instant, and ChatGPT listed in electronic supplementary materials 4 – 6 , respectively. The identified SLOs emphasize the application of pharmacology knowledge within a clinical context, focusing on competencies needed to function independently in early residency stages. These SLOs go beyond knowledge recall and mechanisms of action to encompass competencies related to clinical problem-solving, rational prescribing, and holistic patient management. The SLOs generated require higher cognitive ability of the learner: action verbs such as demonstrate, apply, evaluate, analyze, develop, justify, recommend, interpret, manage, adjust, educate, refer, design, initiate & titrate were frequently used.

The MCQs for the pre-clerkship phase identified by Sage Poe, Claude-Instant, and ChatGPT listed in the electronic supplementary materials 7 – 9 , respectively, and those identified with the search query based on the clinical vignette in electronic supplementary materials ( 10 – 12 ).

All MCQs generated by the AIs in each of the four domains specified [mechanism of action (MOA); pharmacokinetics; adverse drug reactions (ADRs), and indications for antihypertensive drugs] are quality test items with potential content validity. The test items on MOA generated by Sage Poe included themes such as renin-angiotensin-aldosterone (RAAS) system, beta-adrenergic blockers (BB), calcium channel blockers (CCB), potassium channel openers, and centrally acting antihypertensives; on pharmacokinetics included high oral bioavailability/metabolism in liver [angiotensin receptor blocker (ARB)-losartan], long half-life and renal elimination [angiotensin converting enzyme inhibitors (ACEI)-lisinopril], metabolism by both liver and kidney (beta-blocker (BB)-metoprolol], rapid onset- short duration of action (direct vasodilator-hydralazine), and long-acting transdermal drug delivery (centrally acting-clonidine). Regarding the ADR theme, dry cough, angioedema, and hyperkalemia by ACEIs in susceptible patients, reflex tachycardia by CCB/amlodipine, and orthostatic hypotension by CCB/verapamil addressed. Clinical indications included the drug of choice for hypertensive patients with concomitant comorbidity such as diabetics (ACEI-lisinopril), heart failure and low ejection fraction (BB-carvedilol), hypertensive urgency/emergency (alpha cum beta receptor blocker-labetalol), stroke in patients with history recurrent stroke or transient ischemic attack (ARB-losartan), and preeclampsia (methyldopa).

Almost similar themes under each domain were identified by the Claude-Instant AI platform with few notable exceptions: hydrochlorothiazide (instead of clonidine) in MOA and pharmacokinetics domains, respectively; under the ADR domain ankle edema/ amlodipine, sexual dysfunction and fatigue in male due to alpha-1 receptor blocker; under clinical indications the best initial monotherapy for clinical scenarios such as a 55-year old male with Stage-2 hypertension; a 75-year-old man Stage 1 hypertension; a 35-year-old man with Stage I hypertension working on night shifts; and a 40-year-old man with stage 1 hypertension and hyperlipidemia.

As with Claude-Instant AI, ChatGPT-generated test items on MOA were mostly similar. However, under the pharmacokinetic domain, immediate- and extended-release metoprolol, the effect of food to enhance the oral bioavailability of ramipril, and the highest oral bioavailability of amlodipine compared to other commonly used antihypertensives were the themes identified. Whereas the other ADR themes remained similar, constipation due to verapamil was a new theme addressed. Notably, in this test item, amlodipine was an option that increased the difficulty of this test item because amlodipine therapy is also associated with constipation, albeit to a lesser extent, compared to verapamil. In the clinical indication domain, the case description asking “most commonly used in the treatment of hypertension and heart failure” is controversial because the options listed included losartan, ramipril, and hydrochlorothiazide but the suggested correct answer was ramipril. This is a good example to stress the importance of vetting the AI-generated MCQ by experts for content validity and to assure robust psychometrics. The MCQ on the most used drug in the treatment of “hypertension and diabetic nephropathy” is more explicit as opposed to “hypertension and diabetes” by Claude-Instant because the therapeutic concept of reducing or delaying nephropathy must be distinguished from prevention of nephropathy, although either an ACEI or ARB is the drug of choice for both indications.

It is important to align student assessment to the curriculum; in the PBL curriculum, MCQs with a clinical vignette are preferred. The modification of the query specifying the search to generate MCQs with a clinical vignette on domains specified previously gave appropriate output by all three AI platforms evaluated (Sage Poe; Claude- Instant; Chat GPT). The scenarios generated had a good clinical fidelity and educational fit for the pre-clerkship student perspective.

The errors observed with AI outputs on the A-type MCQs are summarized in Table  2 . No significant pattern was observed except that Claude-Instant© generated test items in a stereotyped format such as the same choices for all test items related to pharmacokinetics and indications, and all the test items in the ADR domain are linked to the mechanisms of action of drugs. This illustrates the importance of reviewing AI-generated test items by content experts for content validity to ensure alignment with evidence-based medicine and up-to-date treatment guidelines.

The test items generated by ChatGPT had the advantage of explanations supplied rendering these more useful for learners to support self-study. The following examples illustrate this assertion: “ A patient with hypertension is started on a medication that works by blocking beta-1 receptors in the heart (metoprolol)”. Metoprolol is a beta blocker that works by blocking beta-1 receptors in the heart, which reduces heart rate and cardiac output, resulting in a decrease in blood pressure. However, this explanation is incomplete because there is no mention of other less important mechanisms, of beta receptor blockers on renin release. Also, these MCQs were mostly recall type: Which of the following medications is known to have a significant first-pass effect? The explanation reads: propranolol is known to have a significant first pass-effect, meaning that a large portion of the drug is metabolized by the liver before it reaches systemic circulation. Losartan, amlodipine, ramipril, and hydrochlorothiazide do not have significant first-pass effect. However, it is also important to extend the explanation further by stating that the first-pass effect of propranolol does not lead to total loss of pharmacological activity because the metabolite hydroxy propranolol also has potent beta-blocking activity. Another MCQ test item had a construction defect: “A patient with hypertension is started on a medication that can cause photosensitivity. Which of the following medications is most likely responsible?” Options included: losartan, amlodipine, ramipril, hydrochlorothiazide, hydrochlorothiazide/triamterene. The explanation states that hydrochlorothiazide (HCTZ) can cause photosensitivity, which is increased sensitivity to sunlight that can cause skin rashes and sunburns. Losartan, amlodipine, ramipril, and HCTZ/triamterene are less likely to cause photosensitivity. However, it is irrational to claim that HCTZ/triamterene, one of the wrong options, is less likely to cause photosensitivity. The difficulty index of such test items is likely unacceptable in high-stake tests to make equitable psychometric decisions.

The responses from Sage Poe, Claude-Instant, and ChatGPT on guidance for standard-setting methods are listed in the electronic supplementary materials 13 – 15 . Sage Poe offered this information with the caveat that “the results of this simulation were purely hypothetical and do not reflect the actual performance of any real students”. The simulation estimates of examinees correctly answering test items ranged from 60 to 90% for the mechanism of action, 65–85% for pharmacokinetics, 60–80% for adverse reactions, and 75–90% for clinical indications of antihypertensives.

In contrast, Claude-Instant admitted that “Performing a full simulation with 100 medical students is beyond the scope of what I can do as an AI assistant. I can generate sample MCQ questions and answers, but I do not have access to a population of medical students to administer the questions. A true simulation that administers questions to real students and analyzes the results would require access to a database of medical students, a platform to deliver the questions, and code to analyze the responses. As an AI assistant, I lack these capabilities. While I cannot perform a full simulation, here are some considerations for administering MCQ questions to medical students: provide details on the purpose and instructions for the test to ensure students understand what is expected out of them; consider including some easier questions in addition to more challenging ones to boost students’ confidence and motivation; avoid ambiguous wording and ensure each question has only one unambiguously correct answer option; when analyzing results, look for questions factors that may have impacted performance, like unclear wording or answer options that are too similar. Focusing on weak questions can help improve future tests; compare performance across different student groups to identify areas where more teaching may be needed; provide detailed answer explanations and review sessions after the test to help students learn from any errors and reinforce correct concepts”. However, ChatGPT offered the simulation result estimates for the 20 MCQ on antihypertensive drugs: mechanism of action ranged from 67 to 92%, pharmacokinetics 63–86%, adverse effects 65–82%, and clinical indications 64–89%. Furthermore, it also stated that “Overall, the performance of the students was quite good, with most questions having a response rate of over 70%. However, there were some questions where the projected correct response rate was rather low, such as Question #5 (Mechanism of Action of Hydrochlorothiazide; see Electronic Supplementary Material 12 ) and Question 18 (Indications for Verapamil; see Electronic Supplementary Material 10 ). This may suggest areas where students need more focused education or review.”

We asked AI assistants to generate 20 integrated case cluster MCQs with 2 test items in each cluster with five options for undergraduate medical students in the pre-clerkship phase integrating pharmacology and physiology related to systemic hypertension with a case vignette and the responses by Sage Poe, Claude-Instant, and ChatGPT are listed in the electronic supplementary materials ( 16 – 18 ). In all instances, the test items generated had focused case descriptions in the form of a clinical vignette, and horizontal integration across the pathophysiology of hypertension and pharmacology of antihypertensive drugs. These test items mostly targeted the ‘knows (knowledge)’ or ‘knows how (competence)’ level on Miller’s pyramid and are suitable for assessing the clinical competence of pre-clerkship medical students, especially in an integrated PBL curriculum. Both the AI assistants generated excellent clinical vignettes and themes; however, most of the cluster MCQs by ChatGPT had “None of the above” as an option, which is often considered a test item construction flaw. Notwithstanding these limitations, case cluster integrated test items are valuable for learners to integrate their knowledge of different basic medical sciences and their application to clinical sciences. This integrated approach can be used for both instructional and student assessment purposes to make the course more meaningful. Indeed, one of the basic tenets of PBL is curriculum integration.

In the next query, we asked AI assistants to write integrated case cluster MCQs with 2 test items in each cluster with 5 options for undergraduate medical students at the time of graduation integrating pharmacology and physiology of hypertension with a case vignette. The responses are listed in the electronic supplementary materials 19 – 21 . Claude-Instant has provided only partial responses to both the queries. Overall, the information provided in case vignettes requires the learner to interpret the clinical information and make therapeutic decisions. Such test items are appropriate for assessing the minimum competency of medical school graduates to start internship or residency programs; these MCQs are generally comparable to test items used in USMLE Part 2. The evaluation of the integrated case cluster test items provided by all the AI platforms is summarized in Table  3 . In general, the AI outputs had at least one construction defect, and did not include all the core sub-topics to a sufficient degree.

When asked to simulate the above-generated test items to 100 undergraduate medical students at the time of graduation and provide a percentage of students who are likely to correctly answer each item, the AI assistant responded:

“ As an AI language model, I cannot simulate the responses of 100 undergraduate medical students likely to obtain the correct answer for each question. However, I can provide some general information about the difficulty level of each question based on the complexity of the concepts being tested and the level of integration between pharmacology and physiology ”.

AI assistant went on to clarify that questions that require knowledge of basic pharmacology principles, such as the mechanism of action of specific drugs, are likely to be easier for students to answer correctly. Test items that require an understanding of the physiological mechanisms underlying hypertension and correlating with symptoms are likely to be more challenging for students. The AI assistant sorted these test items into two categories accordingly. Overall, the difficulty level of the test item is based on the level of integration between pharmacology and pathophysiology. Test items that require an understanding of both pharmacological and physiological mechanisms are likely to be more challenging for students requiring a strong foundation in both pharmacology and physiology concepts to be able to correctly answer integrated case-cluster MCQs.

Short answer questions

The responses to a search query on generating SAQs appropriate to the pre-clerkship phase Sage Poe, Claude-Instant, and ChatGPT generated items are listed in the electronic supplementary materials 22 – 24 for difficult questions and 25–27 for moderately difficult questions.

It is apparent from these case vignette descriptions that the short answer question format varied. Accordingly, the scope for asking individual questions for each scenario is open-ended. In all instances, model answers are supplied which are helpful for the course instructor to plan classroom lessons, identify appropriate instructional methods, and establish rubrics for grading the answer scripts, and as a study guide for students.

We then wanted to see to what extent AI can differentiate the difficulty of the SAQ by replacing the search term “difficult” with “moderately difficult” in the above search prompt: the changes in the revised case scenarios are substantial. Perhaps the context of learning and practice (and the level of the student in the MD/medical program) may determine the difficulty level of SAQ generated. It is worth noting that on changing the search from cardiology to internal medicine rotation in Sage Poe the case description also changed. Thus, it is essential to select an appropriate AI assistant, perhaps by trial and error, to generate quality SAQs. Most of the individual questions tested stand-alone knowledge and did not require students to demonstrate integration.

The responses of Sage Poe, Claude-Instant, and ChatGPT for the search query to generate SAQs at the time of graduation are listed in the electronic supplementary materials 28 – 30 . It is interesting to note how AI assistants considered the stage of the learner while generating the SAQ. The response by Sage Poe is illustrative for comparison. “You are a newly graduated medical student who is working in a hospital” versus “You are a medical student in your pre-clerkship.”

Some questions were retained, deleted, or modified to align with competency appropriate to the context (Electronic Supplementary Materials 28 – 30 ). Overall, the test items at both levels from all AI platforms were technically accurate and thorough addressing the topics related to different disciplines (Table  3 ). The differences in learning objective transition are summarized in Table  4 . A comparison of learning objectives revealed that almost all objectives remained the same except for a few (Table  5 ).

A similar trend was apparent with test items generated by other AI assistants, such as ChatGPT. The contrasting differences in questions are illustrated by the vertical integration of basic sciences and clinical sciences (Table  6 ).

Taken together, these in-depth qualitative comparisons suggest that AI assistants such as Sage Poe and ChatGPT consider the learner’s stage of training in designing test items, learning outcomes, and answers expected from the examinee. It is critical to state the search query explicitly to generate quality output by AI assistants.

The OSPE test items generated by Claude-Instant and ChatGPT appropriate to the pre-clerkship phase (without mentioning “appropriate instructions for the patients”) are listed in the electronic supplementary materials 31 and 32 and with patient instructions on the electronic supplementary materials 33 and 34 . For reasons unknown, Sage Poe did not provide any response to this search query.

The five OSPE items generated were suitable to assess the prescription writing competency of pre-clerkship medical students. The clinical scenarios identified by the three AI platforms were comparable; these scenarios include patients with hypertension and impaired glucose tolerance in a 65-year-old male, hypertension with chronic kidney disease (CKD) in a 55-year-old woman, resistant hypertension with obstructive sleep apnea in a 45-year-old man, and gestational hypertension at 32 weeks in a 35-year-old (Claude-Instant AI). Incorporating appropriate instructions facilitates the learner’s ability to educate patients and maximize safe and effective therapy. The OSPE item required students to write a prescription with guidance to start conservatively, choose an appropriate antihypertensive drug class (drug) based on the patients’ profile, specifying drug name, dose, dosing frequency, drug quantity to be dispensed, patient name, date, refill, and caution as appropriate, in addition to prescribers’ name, signature, and license number. In contrast, ChatGPT identified clinical scenarios to include patients with hypertension and CKD, hypertension and bronchial asthma, gestational diabetes, hypertension and heart failure, and hypertension and gout (ChatGPT). Guidance for dosage titration, warnings to be aware, safety monitoring, and frequency of follow-up and dose adjustment. These test items are designed to assess learners’ knowledge of P & T of antihypertensives, as well as their ability to provide appropriate instructions to patients. These clinical scenarios for writing prescriptions assess students’ ability to choose an appropriate drug class, write prescriptions with proper labeling and dosing, reflect drug safety profiles, and risk factors, and make modifications to meet the requirements of special populations. The prescription is required to state the drug name, dose, dosing frequency, patient name, date, refills, and cautions or instructions as needed. A conservative starting dose, once or twice daily dosing frequency based on the drug, and instructions to titrate the dose slowly if required.

The responses from Claude-Instant and ChatGPT for the search query related to generating OSPE test items at the time of graduation are listed in electronic supplementary materials 35 and 36 . In contrast to the pre-clerkship phase, OSPEs generated for graduating doctors’ competence assessed more advanced drug therapy comprehension. For example, writing a prescription for:

(1) A 65-year- old male with resistant hypertension and CKD stage 3 to optimize antihypertensive regimen required the answer to include starting ACEI and diuretic, titrating the dosage over two weeks, considering adding spironolactone or substituting ACEI with an ARB, and need to closely monitor serum electrolytes and kidney function closely.

(2) A 55-year-old woman with hypertension and paroxysmal arrhythmia required the answer to include switching ACEI to ARB due to cough, adding a CCB or beta blocker for rate control needs, and adjusting the dosage slowly and monitoring for side effects.

(3) A 45-year-old man with masked hypertension and obstructive sleep apnea require adding a centrally acting antihypertensive at bedtime and increasing dosage as needed based on home blood pressure monitoring and refer to CPAP if not already using one.

(4) A 75-year-old woman with isolated systolic hypertension and autonomic dysfunction to require stopping diuretic and switching to an alpha blocker, upward dosage adjustment and combining with other antihypertensives as needed based on postural blood pressure changes and symptoms.

(5) A 35-year-old pregnant woman with preeclampsia at 29 weeks require doubling methyldopa dose and consider adding labetalol or nifedipine based on severity and educate on signs of worsening and to follow-up immediately for any concerning symptoms.

These case scenarios are designed to assess the ability of the learner to comprehend the complexity of antihypertensive regimens, make evidence-based regimen adjustments, prescribe multidrug combinations based on therapeutic response and tolerability, monitor complex patients for complications, and educate patients about warning signs and follow-up.

A similar output was provided by ChatGPT, with clinical scenarios such as prescribing for patients with hypertension and myocardial infarction; hypertension and chronic obstructive pulmonary airway disease (COPD); hypertension and a history of angina; hypertension and a history of stroke, and hypertension and advanced renal failure. In these cases, wherever appropriate, pharmacotherapeutic issues like taking ramipril after food to reduce side effects such as giddiness; selection of the most appropriate beta-blocker such as nebivolol in patients with COPD comorbidity; the importance of taking amlodipine at the same time every day with or without food; preference for telmisartan among other ARBs in stroke; choosing furosemide in patients with hypertension and edema and taking the medication with food to reduce the risk of gastrointestinal adverse effect are stressed.

The AI outputs on OSPE test times were observed to be technically accurate, thorough in addressing core sub-topics suitable for the learner’s level and did not have any construction defects (Table  3 ). Both AIs provided the model answers with explanatory notes. This facilitates the use of such OSPEs for self-assessment by learners for formative assessment purposes. The detailed instructions are helpful in creating optimized therapy regimens, and designing evidence-based regimens, to provide appropriate instructions to patients with complex medical histories. One can rely on multiple AI sources to identify, shortlist required case scenarios, and OSPE items, and seek guidance on expected model answers with explanations. The model answer guidance for antihypertensive drug classes is more appropriate (rather than a specific drug of a given class) from a teaching/learning perspective. We believe that these scenarios can be refined further by providing a focused case history along with relevant clinical and laboratory data to enhance clinical fidelity and bring a closer fit to the competency framework.

In the present study, AI tools have generated SLOs that comply with the current principles of medical education [ 15 ]. AI tools are valuable in constructing SLOs and so are especially useful for medical fraternities where training in medical education is perceived as inadequate, more so in the early stages of their academic career. Data suggests that only a third of academics in medical schools have formal training in medical education [ 16 ] which is a limitation. Thus, the credibility of alternatives, such as the AIs, is evaluated to generate appropriate course learning outcomes.

We observed that the AI platforms in the present study generated quality test items suitable for different types of assessment purposes. The AI-generated outputs were similar with minor variation. We have used generative AIs in the present study that could generate new content from their training dataset [ 17 ]. Problem-based and interactive learning approaches are referred to as “bottom-up” where learners obtain first-hand experience in solving the cases first and then indulge in discussion with the educators to refine their understanding and critical thinking skills [ 18 ]. We suggest that AI tools can be useful for this approach for imparting the core knowledge and skills related to Pharmacology and Therapeutics to undergraduate medical students. A recent scoping review evaluating the barriers to writing quality test items based on 13 studies has concluded that motivation, time constraints, and scheduling were the most common [ 19 ]. AI tools can be valuable considering the quick generation of quality test items and time management. However, as observed in the present study, the AI-generated test items nevertheless require scrutiny by faculty members for content validity. Moreover, it is important to train faculty in AI technology-assisted teaching and learning. The General Medical Council recommends taking every opportunity to raise the profile of teaching in medical schools [ 20 ]. Hence, both the academic faculty and the institution must consider investing resources in AI training to ensure appropriate use of the technology [ 21 ].

The AI outputs assessed in the present study had errors, particularly with A-type MCQs. One notable observation was that often the AI tools were unable to differentiate the differences between ACEIs and ARBs. AI platforms access several structured and unstructured data, in addition to images, audio, and videos. Hence, the AI platforms can commit errors due to extracting details from unauthenticated sources [ 22 ] created a framework identifying 28 factors for reconstructing the path of AI failures and for determining corrective actions. This is an area of interest for AI technical experts to explore. Also, this further iterates the need for human examination of test items before using them for assessment purposes.

There are concerns that AIs can memorize and provide answers from their training dataset, which they are not supposed to do [ 23 ]. Hence, the use of AIs-generated test items for summative examinations is debatable. It is essential to ensure and enhance the security features of AI tools to reduce or eliminate cross-contamination of test items. Researchers have emphasized that AI tools will only reach their potential if developers and users can access full-text non-PDF formats that help machines comprehend research papers and generate the output [ 24 ].

AI platforms may not always have access to all standard treatment guidelines. However, in the present study, it was observed that all three AI platforms generally provided appropriate test items regarding the choice of medications, aligning with recommendations from contemporary guidelines and standard textbooks in pharmacology and therapeutics. The prompts used in the study were specifically focused on the pre-clerkship phase of the undergraduate medical curriculum (and at the time of their graduation) and assessed fundamental core concepts, which were also reflected in the AI outputs. Additionally, the recommended first-line antihypertensive drug classes have been established for several decades, and information regarding their pharmacokinetics, ADRs, and indications is well-documented in the literature.

Different paradigms and learning theories have been proposed to support AI in education. These paradigms include AI- directed (learner as recipient), AI-supported (learner as collaborator), and AI-empowered (learner as leader) that are based on Behaviorism, Cognitive-Social constructivism, and Connectivism-Complex adaptive systems, respectively [ 25 ]. AI techniques have potential to stimulate and advance instructional and learning sciences. More recently a three- level model that synthesizes and unifies existing learning theories to model the roles of AIs in promoting learning process has been proposed [ 26 ]. The different components of our study rely upon these paradigms and learning theories as the theoretical underpinning.

Strengths and limitations

To the best of our knowledge, this is the first study evaluating the utility of AI platforms in generating test items related to a discipline in the undergraduate medical curriculum. We have evaluated the AI’s ability to generate outputs related to most types of assessment in the undergraduate medical curriculum. The key lessons learnt for improving the AI-generated test item quality from the present study are outlined in Table  7 . We used a structured framework for assessing the content validity of the test items. However, we have demonstrated using a single case study (hypertension) as a pilot experiment. We chose to evaluate anti-hypertensive drugs as it is a core learning objective and one of the most common disorders relevant to undergraduate medical curricula worldwide. It would be interesting to explore the output from AI platforms for other common (and uncommon/region-specific) disorders, non-/semi-core objectives, and disciplines other than Pharmacology and Therapeutics. An area of interest would be to look at the content validity of the test items generated for different curricula (such as problem-based, integrated, case-based, and competency-based) during different stages of the learning process. Also, we did not attempt to evaluate the generation of flowcharts, algorithms, or figures for generating test items. Another potential area for exploring the utility of AIs in medical education would be repeated procedural practices such as the administration of drugs through different routes by trainee residents [ 27 ]. Several AI tools have been identified for potential application in enhancing classroom instructions and assessment purposes pending validation in prospective studies [ 28 ]. Lastly, we did not administer the AI-generated test items to students and assessed their performance and so could not comment on the validity of test item discrimination and difficulty indices. Additionally, there is a need to confirm the generalizability of the findings to other complex areas in the same discipline as well as in other disciplines that pave way for future studies. The conceptual framework used in the present study for evaluating the AI-generated test items needs to be validated in a larger population. Future studies may also try to evaluate the variations in the AI outputs with repetition of the same queries.

Notwithstanding ongoing discussions and controversies, AI tools are potentially useful adjuncts to optimize instructional methods, test blueprinting, test item generation, and guidance for test standard-setting appropriate to learners’ stage in the medical program. However, experts need to critically review the content validity of AI-generated output. These challenges and caveats are to be addressed before the use of widespread use of AIs in medical education can be advocated.

Data availability

All the data included in this study are provided as Electronic Supplementary Materials.

Tolsgaard MG, Pusic MV, Sebok-Syer SS, Gin B, Svendsen MB, Syer MD, Brydges R, Cuddy MM, Boscardin CK. The fundamentals of Artificial Intelligence in medical education research: AMEE Guide 156. Med Teach. 2023;45(6):565–73.

Article   Google Scholar  

Sriwastwa A, Ravi P, Emmert A, Chokshi S, Kondor S, Dhal K, Patel P, Chepelev LL, Rybicki FJ, Gupta R. Generative AI for medical 3D printing: a comparison of ChatGPT outputs to reference standard education. 3D Print Med. 2023;9(1):21.

Azer SA, Guerrero APS. The challenges imposed by artificial intelligence: are we ready in medical education? BMC Med Educ. 2023;23(1):680.

Masters K. Ethical use of Artificial Intelligence in Health Professions Education: AMEE Guide 158. Med Teach. 2023;45(6):574–84.

Nagi F, Salih R, Alzubaidi M, Shah H, Alam T, Shah Z, Househ M. Applications of Artificial Intelligence (AI) in Medical Education: a scoping review. Stud Health Technol Inf. 2023;305:648–51.

Google Scholar  

Mehta N, Harish V, Bilimoria K, et al. Knowledge and attitudes on artificial intelligence in healthcare: a provincial survey study of medical students. MedEdPublish. 2021;10(1):75.

Mir MM, Mir GM, Raina NT, Mir SM, Mir SM, Miskeen E, Alharthi MH, Alamri MMS. Application of Artificial Intelligence in Medical Education: current scenario and future perspectives. J Adv Med Educ Prof. 2023;11(3):133–40.

Garg T. Artificial Intelligence in Medical Education. Am J Med. 2020;133(2):e68.

Matheny ME, Whicher D, Thadaney IS. Artificial intelligence in health care: a report from the National Academy of Medicine. JAMA. 2020;323(6):509–10.

Sage Poe. Available at: https://poe.com/Assistant (Accessed on. 3rd June 2023).

Claude-Instant: Available at: https://poe.com/Claude-instant (Accessed on 3rd. June 2023).

ChatGPT: Available at: https://poe.com/ChatGPT (Accessed on 3rd. June 2023).

James PA, Oparil S, Carter BL, Cushman WC, Dennison-Himmelfarb C, Handler J, Lackland DT, LeFevre ML, MacKenzie TD, Ogedegbe O, Smith SC Jr, Svetkey LP, Taler SJ, Townsend RR, Wright JT Jr, Narva AS, Ortiz E. 2014 evidence-based guideline for the management of high blood pressure in adults: report from the panel members appointed to the Eighth Joint National Committee (JNC 8). JAMA. 2014;311(5):507–20.

Eschenhagen T. Treatment of hypertension. In: Brunton LL, Knollmann BC, editors. Goodman & Gilman’s the pharmacological basis of therapeutics. 14th ed. New York: McGraw Hill; 2023.

Shabatura J. September. Using Bloom’s taxonomy to write effective learning outcomes. https://tips.uark.edu/using-blooms-taxonomy/ (Accessed on 19th 2023).

Trainor A, Richards JB. Training medical educators to teach: bridging the gap between perception and reality. Isr J Health Policy Res. 2021;10(1):75.

Boscardin C, Gin B, Golde PB, Hauer KE. ChatGPT and generative artificial intelligence for medical education: potential and opportunity. Acad Med. 2023. https://doi.org/10.1097/ACM.0000000000005439 . (Published ahead of print).

Duong MT, Rauschecker AM, Rudie JD, Chen PH, Cook TS, Bryan RN, Mohan S. Artificial intelligence for precision education in radiology. Br J Radiol. 2019;92(1103):20190389.

Karthikeyan S, O’Connor E, Hu W. Barriers and facilitators to writing quality items for medical school assessments - a scoping review. BMC Med Educ. 2019;19(1):123.

Developing teachers and trainers in undergraduate medical education. Advice supplementary to Tomorrow’s Doctors. (2009). https://www.gmc-uk.org/-/media/documents/Developing_teachers_and_trainers_in_undergraduate_medical_education___guidance_0815.pdf_56440721.pdf (Accessed on 19th September 2023).

Cooper A, Rodman A. AI and Medical Education - A 21st-Century Pandora’s Box. N Engl J Med. 2023;389(5):385–7.

Chanda SS, Banerjee DN. Omission and commission errors underlying AI failures. AI Soc. 2022;17:1–24.

Narayanan A, Kapoor S. ‘GPT-4 and Professional Benchmarks: The Wrong Answer to the Wrong Question’. Substack newsletter. AI Snake Oil (blog). https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks (Accessed on 19th September 2023).

Brainard J. November. As scientists face a flood of papers, AI developers aim to help. Science, 21 2023. doi.10.1126/science.adn0669.

Ouyang F, Jiao P. Artificial intelligence in education: the three paradigms. Computers Education: Artif Intell. 2021;2:100020.

Gibson D, Kovanovic V, Ifenthaler D, Dexter S, Feng S. Learning theories for artificial intelligence promoting learning processes. Br J Edu Technol. 2023;54(5):1125–46.

Guerrero DT, Asaad M, Rajesh A, Hassan A, Butler CE. Advancing Surgical Education: the Use of Artificial Intelligence in Surgical Training. Am Surg. 2023;89(1):49–54.

Lee S. AI tools for educators. EIT InnoEnergy Master School Teachers Conference. 2023. https://www.slideshare.net/ignatia/ai-toolkit-for-educators?from_action=save (Accessed on 24th September 2023).

Download references

Author information

Authors and affiliations.

Department of Pharmacology & Therapeutics, College of Medicine & Medical Sciences, Arabian Gulf University, Manama, Kingdom of Bahrain

Kannan Sridharan & Reginald P. Sequeira

You can also search for this author in PubMed   Google Scholar

Contributions

RPS– Conceived the idea; KS– Data collection and curation; RPS and KS– Data analysis; RPS and KS– wrote the first draft and were involved in all the revisions.

Corresponding author

Correspondence to Kannan Sridharan .

Ethics declarations

Ethics approval and consent to participate.

Not applicable as neither there was any interaction with humans, nor any personal data was collected in this research study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Sridharan, K., Sequeira, R.P. Artificial intelligence and medical education: application in classroom instruction and student assessment using a pharmacology & therapeutics case study. BMC Med Educ 24 , 431 (2024). https://doi.org/10.1186/s12909-024-05365-7

Download citation

Received : 26 September 2023

Accepted : 28 March 2024

Published : 22 April 2024

DOI : https://doi.org/10.1186/s12909-024-05365-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Medical education
• Pharmacology
• Therapeutics

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]