research and development design

• Search Search Please fill out this field.
• Business Essentials

What Is Research and Development (R&D)?

Investopedia / Ellen Lindner

Research and development (R&D) is the series of activities that companies undertake to innovate. R&D is often the first stage in the development process that results in market research product development, and product testing.

Key Takeaways

• Research and development represents the activities companies undertake to innovate and introduce new products and services or to improve their existing offerings.
• R&D allows a company to stay ahead of its competition by catering to new wants or needs in the market.
• Companies in different sectors and industries conduct R&D—pharmaceuticals, semiconductors, and technology companies generally spend the most.
• R&D is often a broad approach to exploratory advancement, while applied research is more geared towards researching a more narrow scope.
• The accounting for treatment for R&D costs can materially impact a company's income statement and balance sheet.

Understanding Research and Development (R&D)

The concept of research and development is widely linked to innovation both in the corporate and government sectors. R&D allows a company to stay ahead of its competition. Without an R&D program, a company may not survive on its own and may have to rely on other ways to innovate such as engaging in mergers and acquisitions (M&A) or partnerships. Through R&D, companies can design new products and improve their existing offerings.

R&D is distinct from most operational activities performed by a corporation. The research and/or development is typically not performed with the expectation of immediate profit. Instead, it is expected to contribute to the long-term profitability of a company. R&D may often allow companies to secure intellectual property, including patents , copyrights, and trademarks as discoveries are made and products created.

Companies that set up and employ departments dedicated entirely to R&D commit substantial capital to the effort. They must estimate the risk-adjusted return on their R&D expenditures, which inevitably involves risk of capital. That's because there is no immediate payoff, and the return on investment (ROI) is uncertain. As more money is invested in R&D, the level of capital risk increases. Other companies may choose to outsource their R&D for a variety of reasons including size and cost.

Companies across all sectors and industries undergo R&D activities. Corporations experience growth through these improvements and the development of new goods and services. Pharmaceuticals, semiconductors , and software/technology companies tend to spend the most on R&D. In Europe, R&D is known as research and technical or technological development.

Many small and mid-sized businesses may choose to outsource their R&D efforts because they don't have the right staff in-house to meet their needs.

Types of Research and Development (R&D)

There are several different types of R&D that exist in the corporate world and within government. The type used depends entirely on the entity undertaking it and the results can differ.

Basic Research

There are business incubators and accelerators, where corporations invest in startups and provide funding assistance and guidance to entrepreneurs in the hope that innovations will result that they can use to their benefit.

M&As and partnerships are also forms of R&D as companies join forces to take advantage of other companies' institutional knowledge and talent.

Applied Research

One R&D model is a department staffed primarily by engineers who develop new products —a task that typically involves extensive research. There is no specific goal or application in mind with this model. Instead, the research is done for the sake of research.

Development Research

This model involves a department composed of industrial scientists or researchers, all of who are tasked with applied research in technical, scientific, or industrial fields. This model facilitates the development of future products or the improvement of current products and/or operating procedures.

The largest companies may also be the ones that drive the most R&D spend. For example, Amazon has reported $1.147 billion of research and development value on its 2023 annual report.

Advantages and Disadvantages of R&D

There are several key benefits to research and development. It facilitates innovation, allowing companies to improve existing products and services or by letting them develop new ones to bring to the market.

Because R&D also is a key component of innovation, it requires a greater degree of skill from employees who take part. This allows companies to expand their talent pool, which often comes with special skill sets.

The advantages go beyond corporations. Consumers stand to benefit from R&D because it gives them better, high-quality products and services as well as a wider range of options. Corporations can, therefore, rely on consumers to remain loyal to their brands. It also helps drive productivity and economic growth.

Disadvantages

One of the major drawbacks to R&D is the cost. First, there is the financial expense as it requires a significant investment of cash upfront. This can include setting up a separate R&D department, hiring talent, and product and service testing, among others.

Innovation doesn't happen overnight so there is also a time factor to consider. This means that it takes a lot of time to bring products and services to market from conception to production to delivery.

Because it does take time to go from concept to product, companies stand the risk of being at the mercy of changing market trends . So what they thought may be a great seller at one time may reach the market too late and not fly off the shelves once it's ready.

Facilitates innovation

Improved or new products and services

Expands knowledge and talent pool

Increased consumer choice and brand loyalty

Economic driver

Financial investment

Shifting market trends

R&D Accounting

R&D may be beneficial to a company's bottom line, but it is considered an expense . After all, companies spend substantial amounts on research and trying to develop new products and services. As such, these expenses are often reported for accounting purposes on the income statement and do not carry long-term value.

There are certain situations where R&D costs are capitalized and reported on the balance sheet. Some examples include but are not limited to:

• Materials, fixed assets, or other assets have alternative future uses with an estimable value and useful life.
• Software that can be converted or applied elsewhere in the company to have a useful life beyond a specific single R&D project.
• Indirect costs or overhead expenses allocated between projects.
• R&D purchased from a third party that is accompanied by intangible value. That intangible asset may be recorded as a separate balance sheet asset.

R&D Considerations

Before taking on the task of research and development, it's important for companies and governments to consider some of the key factors associated with it. Some of the most notable considerations are:

• Objectives and Outcome: One of the most important factors to consider is the intended goals of the R&D project. Is it to innovate and fill a need for certain products that aren't being sold? Or is it to make improvements on existing ones? Whatever the reason, it's always important to note that there should be some flexibility as things can change over time.
• Timing: R&D requires a lot of time. This involves reviewing the market to see where there may be a lack of certain products and services or finding ways to improve on those that are already on the shelves.
• Cost: R&D costs a great deal of money, especially when it comes to the upfront costs. And there may be higher costs associated with the conception and production of new products rather than updating existing ones.
• Risks: As with any venture, R&D does come with risks. R&D doesn't come with any guarantees, no matter the time and money that goes into it. This means that companies and governments may sacrifice their ROI if the end product isn't successful.

Research and Development vs. Applied Research

Basic research is aimed at a fuller, more complete understanding of the fundamental aspects of a concept or phenomenon. This understanding is generally the first step in R&D. These activities provide a basis of information without directed applications toward products, policies, or operational processes .

Applied research entails the activities used to gain knowledge with a specific goal in mind. The activities may be to determine and develop new products, policies, or operational processes. While basic research is time-consuming, applied research is painstaking and more costly because of its detailed and complex nature.

R&D Tax Credits

The IRS offers a R&D tax credit to encourage innovation and significantly reduction their tax liability. The credit calls for specific types of spend such as product development, process improvement, and software creation.

Enacted under Section 41 of the Internal Revenue Code, this credit encourages innovation by providing a dollar-for-dollar reduction in tax obligations. The eligibility criteria, expanded by the Protecting Americans from Tax Hikes (PATH) Act of 2015, now encompass a broader spectrum of businesses. The credit tens to benefit small-to-midsize enterprises.

To claim R&D tax credits, businesses must document their qualifying expenses and complete IRS Form 6765 (Credit for Increasing Research Activities). The credit, typically ranging from 6% to 8% of annual qualifying expenses, offers businesses a direct offset against federal income tax liabilities. Additionally, businesses can claim up to $250,000 per year against their payroll taxes.

Example of Research and Development (R&D)

One of the more innovative companies of this millennium is Apple Inc. As part of its annual reporting, it has the following to say about its research and development spend:

In 2023, Apple reported having spent $29.915 billion. This is 8% of their annual total net sales. Note that Apple's R&D spend was reported to be higher than the company's selling, general and administrative costs (of $24.932 billion).

Note that the company doesn't go into length about what exactly the R&D spend is for. According to the notes, the company's year-over-year growth was "driven primarily by increases in headcount-related expenses". However, this does not explain the underlying basis carried from prior years (i.e. materials, patents, etc.).

What Is Research and Development?

Research and development refers to the systematic process of investigating, experimenting, and innovating to create new products, processes, or technologies. It encompasses activities such as scientific research, technological development, and experimentation conducted to achieve specific objectives to bring new items to market.

What Types of Activities Can Be Found in Research and Development?

Research and development activities focus on the innovation of new products or services in a company. Among the primary purposes of R&D activities is for a company to remain competitive as it produces products that advance and elevate its current product line. Since R&D typically operates on a longer-term horizon, its activities are not anticipated to generate immediate returns. However, in time, R&D projects may lead to patents, trademarks, or breakthrough discoveries with lasting benefits to the company. 

Why Is Research and Development Important?

Given the rapid rate of technological advancement, R&D is important for companies to stay competitive. Specifically, R&D allows companies to create products that are difficult for their competitors to replicate. Meanwhile, R&D efforts can lead to improved productivity that helps increase margins, further creating an edge in outpacing competitors. From a broader perspective, R&D can allow a company to stay ahead of the curve, anticipating customer demands or trends.

The Bottom Line

There are many things companies can do in order to advance in their industries and the overall market. Research and development is just one way they can set themselves apart from their competition. It opens up the potential for innovation and increasing sales. However, it does come with some drawbacks—the most obvious being the financial cost and the time it takes to innovate.

Amazon. " 2023 Annual Report ."

Internal Revenue Service. " Research Credit ."

Internal Revenue Service. " About Form 6765, Credit for Increasing Research Activities ."

Apple. " 2023 Annual Report ."

• Terms of Service
• Editorial Policy
• Privacy Policy
• Your Privacy Choices

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Methodology

Research Design | Step-by-Step Guide with Examples

Published on 5 May 2022 by Shona McCombes . Revised on 20 March 2023.

A research design is a strategy for answering your research question  using empirical data. Creating a research design means making decisions about:

• Your overall aims and approach
• The type of research design you’ll use
• Your sampling methods or criteria for selecting subjects
• Your data collection methods
• The procedures you’ll follow to collect data
• Your data analysis methods

A well-planned research design helps ensure that your methods match your research aims and that you use the right kind of analysis for your data.

Table of contents

Step 1: consider your aims and approach, step 2: choose a type of research design, step 3: identify your population and sampling method, step 4: choose your data collection methods, step 5: plan your data collection procedures, step 6: decide on your data analysis strategies, frequently asked questions.

• Introduction

Before you can start designing your research, you should already have a clear idea of the research question you want to investigate.

There are many different ways you could go about answering this question. Your research design choices should be driven by your aims and priorities – start by thinking carefully about what you want to achieve.

The first choice you need to make is whether you’ll take a qualitative or quantitative approach.

Qualitative research designs tend to be more flexible and inductive , allowing you to adjust your approach based on what you find throughout the research process.

Quantitative research designs tend to be more fixed and deductive , with variables and hypotheses clearly defined in advance of data collection.

It’s also possible to use a mixed methods design that integrates aspects of both approaches. By combining qualitative and quantitative insights, you can gain a more complete picture of the problem you’re studying and strengthen the credibility of your conclusions.

Practical and ethical considerations when designing research

As well as scientific considerations, you need to think practically when designing your research. If your research involves people or animals, you also need to consider research ethics .

• How much time do you have to collect data and write up the research?
• Will you be able to gain access to the data you need (e.g., by travelling to a specific location or contacting specific people)?
• Do you have the necessary research skills (e.g., statistical analysis or interview techniques)?
• Will you need ethical approval ?

At each stage of the research design process, make sure that your choices are practically feasible.

Prevent plagiarism, run a free check.

Within both qualitative and quantitative approaches, there are several types of research design to choose from. Each type provides a framework for the overall shape of your research.

Types of quantitative research designs

Quantitative designs can be split into four main types. Experimental and   quasi-experimental designs allow you to test cause-and-effect relationships, while descriptive and correlational designs allow you to measure variables and describe relationships between them.

With descriptive and correlational designs, you can get a clear picture of characteristics, trends, and relationships as they exist in the real world. However, you can’t draw conclusions about cause and effect (because correlation doesn’t imply causation ).

Experiments are the strongest way to test cause-and-effect relationships without the risk of other variables influencing the results. However, their controlled conditions may not always reflect how things work in the real world. They’re often also more difficult and expensive to implement.

Types of qualitative research designs

Qualitative designs are less strictly defined. This approach is about gaining a rich, detailed understanding of a specific context or phenomenon, and you can often be more creative and flexible in designing your research.

The table below shows some common types of qualitative design. They often have similar approaches in terms of data collection, but focus on different aspects when analysing the data.

Your research design should clearly define who or what your research will focus on, and how you’ll go about choosing your participants or subjects.

In research, a population is the entire group that you want to draw conclusions about, while a sample is the smaller group of individuals you’ll actually collect data from.

Defining the population

A population can be made up of anything you want to study – plants, animals, organisations, texts, countries, etc. In the social sciences, it most often refers to a group of people.

For example, will you focus on people from a specific demographic, region, or background? Are you interested in people with a certain job or medical condition, or users of a particular product?

The more precisely you define your population, the easier it will be to gather a representative sample.

Sampling methods

Even with a narrowly defined population, it’s rarely possible to collect data from every individual. Instead, you’ll collect data from a sample.

To select a sample, there are two main approaches: probability sampling and non-probability sampling . The sampling method you use affects how confidently you can generalise your results to the population as a whole.

Probability sampling is the most statistically valid option, but it’s often difficult to achieve unless you’re dealing with a very small and accessible population.

For practical reasons, many studies use non-probability sampling, but it’s important to be aware of the limitations and carefully consider potential biases. You should always make an effort to gather a sample that’s as representative as possible of the population.

Case selection in qualitative research

In some types of qualitative designs, sampling may not be relevant.

For example, in an ethnography or a case study, your aim is to deeply understand a specific context, not to generalise to a population. Instead of sampling, you may simply aim to collect as much data as possible about the context you are studying.

In these types of design, you still have to carefully consider your choice of case or community. You should have a clear rationale for why this particular case is suitable for answering your research question.

For example, you might choose a case study that reveals an unusual or neglected aspect of your research problem, or you might choose several very similar or very different cases in order to compare them.

Data collection methods are ways of directly measuring variables and gathering information. They allow you to gain first-hand knowledge and original insights into your research problem.

You can choose just one data collection method, or use several methods in the same study.

Survey methods

Surveys allow you to collect data about opinions, behaviours, experiences, and characteristics by asking people directly. There are two main survey methods to choose from: questionnaires and interviews.

Observation methods

Observations allow you to collect data unobtrusively, observing characteristics, behaviours, or social interactions without relying on self-reporting.

Observations may be conducted in real time, taking notes as you observe, or you might make audiovisual recordings for later analysis. They can be qualitative or quantitative.

Other methods of data collection

There are many other ways you might collect data depending on your field and topic.

If you’re not sure which methods will work best for your research design, try reading some papers in your field to see what data collection methods they used.

Secondary data

If you don’t have the time or resources to collect data from the population you’re interested in, you can also choose to use secondary data that other researchers already collected – for example, datasets from government surveys or previous studies on your topic.

With this raw data, you can do your own analysis to answer new research questions that weren’t addressed by the original study.

Using secondary data can expand the scope of your research, as you may be able to access much larger and more varied samples than you could collect yourself.

However, it also means you don’t have any control over which variables to measure or how to measure them, so the conclusions you can draw may be limited.

As well as deciding on your methods, you need to plan exactly how you’ll use these methods to collect data that’s consistent, accurate, and unbiased.

Planning systematic procedures is especially important in quantitative research, where you need to precisely define your variables and ensure your measurements are reliable and valid.

Operationalisation

Some variables, like height or age, are easily measured. But often you’ll be dealing with more abstract concepts, like satisfaction, anxiety, or competence. Operationalisation means turning these fuzzy ideas into measurable indicators.

If you’re using observations , which events or actions will you count?

If you’re using surveys , which questions will you ask and what range of responses will be offered?

You may also choose to use or adapt existing materials designed to measure the concept you’re interested in – for example, questionnaires or inventories whose reliability and validity has already been established.

Reliability and validity

Reliability means your results can be consistently reproduced , while validity means that you’re actually measuring the concept you’re interested in.

For valid and reliable results, your measurement materials should be thoroughly researched and carefully designed. Plan your procedures to make sure you carry out the same steps in the same way for each participant.

If you’re developing a new questionnaire or other instrument to measure a specific concept, running a pilot study allows you to check its validity and reliability in advance.

Sampling procedures

As well as choosing an appropriate sampling method, you need a concrete plan for how you’ll actually contact and recruit your selected sample.

That means making decisions about things like:

• How many participants do you need for an adequate sample size?
• What inclusion and exclusion criteria will you use to identify eligible participants?
• How will you contact your sample – by mail, online, by phone, or in person?

If you’re using a probability sampling method, it’s important that everyone who is randomly selected actually participates in the study. How will you ensure a high response rate?

If you’re using a non-probability method, how will you avoid bias and ensure a representative sample?

Data management

It’s also important to create a data management plan for organising and storing your data.

Will you need to transcribe interviews or perform data entry for observations? You should anonymise and safeguard any sensitive data, and make sure it’s backed up regularly.

Keeping your data well organised will save time when it comes to analysing them. It can also help other researchers validate and add to your findings.

On their own, raw data can’t answer your research question. The last step of designing your research is planning how you’ll analyse the data.

Quantitative data analysis

In quantitative research, you’ll most likely use some form of statistical analysis . With statistics, you can summarise your sample data, make estimates, and test hypotheses.

Using descriptive statistics , you can summarise your sample data in terms of:

• The distribution of the data (e.g., the frequency of each score on a test)
• The central tendency of the data (e.g., the mean to describe the average score)
• The variability of the data (e.g., the standard deviation to describe how spread out the scores are)

The specific calculations you can do depend on the level of measurement of your variables.

Using inferential statistics , you can:

• Make estimates about the population based on your sample data.
• Test hypotheses about a relationship between variables.

Regression and correlation tests look for associations between two or more variables, while comparison tests (such as t tests and ANOVAs ) look for differences in the outcomes of different groups.

Your choice of statistical test depends on various aspects of your research design, including the types of variables you’re dealing with and the distribution of your data.

Qualitative data analysis

In qualitative research, your data will usually be very dense with information and ideas. Instead of summing it up in numbers, you’ll need to comb through the data in detail, interpret its meanings, identify patterns, and extract the parts that are most relevant to your research question.

Two of the most common approaches to doing this are thematic analysis and discourse analysis .

There are many other ways of analysing qualitative data depending on the aims of your research. To get a sense of potential approaches, try reading some qualitative research papers in your field.

A sample is a subset of individuals from a larger population. Sampling means selecting the group that you will actually collect data from in your research.

For example, if you are researching the opinions of students in your university, you could survey a sample of 100 students.

Statistical sampling allows you to test a hypothesis about the characteristics of a population. There are various sampling methods you can use to ensure that your sample is representative of the population as a whole.

Operationalisation means turning abstract conceptual ideas into measurable observations.

For example, the concept of social anxiety isn’t directly observable, but it can be operationally defined in terms of self-rating scores, behavioural avoidance of crowded places, or physical anxiety symptoms in social situations.

Before collecting data , it’s important to consider how you will operationalise the variables that you want to measure.

The research methods you use depend on the type of data you need to answer your research question .

• If you want to measure something or test a hypothesis , use quantitative methods . If you want to explore ideas, thoughts, and meanings, use qualitative methods .
• If you want to analyse a large amount of readily available data, use secondary data. If you want data specific to your purposes with control over how they are generated, collect primary data.
• If you want to establish cause-and-effect relationships between variables , use experimental methods. If you want to understand the characteristics of a research subject, use descriptive methods.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2023, March 20). Research Design | Step-by-Step Guide with Examples. Scribbr. Retrieved 21 May 2024, from https://www.scribbr.co.uk/research-methods/research-design/

Is this article helpful?

Shona McCombes

Building an R&D strategy for modern times

The global investment in research and development (R&D) is staggering. In 2019 alone, organizations around the world spent $2.3 trillion on R&D—the equivalent of roughly 2 percent of global GDP—about half of which came from industry and the remainder from governments and academic institutions. What’s more, that annual investment has been growing at approximately 4 percent per year over the past decade. 1 2.3 trillion on purchasing-power-parity basis; 2019 global R&D funding forecast , Supplement, R&D Magazine, March 2019, rdworldonline.com.

While the pharmaceutical sector garners much attention due to its high R&D spending as a percentage of revenues, a comparison based on industry profits shows that several industries, ranging from high tech to automotive to consumer, are putting more than 20 percent of earnings before interest, taxes, depreciation, and amortization (EBITDA) back into innovation research (Exhibit 1).

What do organizations expect to get in return? At the core, they hope their R&D investments yield the critical technology from which they can develop new products, services, and business models. But for R&D to deliver genuine value, its role must be woven centrally into the organization’s mission. R&D should help to both deliver and shape corporate strategy, so that it develops differentiated offerings for the company’s priority markets and reveals strategic options, highlighting promising ways to reposition the business through new platforms and disruptive breakthroughs.

Yet many enterprises lack an R&D strategy that has the necessary clarity, agility, and conviction to realize the organization’s aspirations. Instead of serving as the company’s innovation engine, R&D ends up isolated from corporate priorities, disconnected from market developments, and out of sync with the speed of business. Amid a growing gap in performance  between those that innovate successfully and those that do not, companies wishing to get ahead and stay ahead of competitors need a robust R&D strategy that makes the most of their innovation investments. Building such a strategy takes three steps: understanding the challenges that often work as barriers to R&D success, choosing the right ingredients for your strategy, and then pressure testing it before enacting it.

Overcoming the barriers to successful R&D

The first step to building an R&D strategy is to understand the four main challenges that modern R&D organizations face:

Innovation cycles are accelerating. The growing reliance on software and the availability of simulation and automation technologies have caused the cost of experimentation to plummet while raising R&D throughput. The pace of corporate innovation is further spurred by the increasing emergence of broadly applicable technologies, such as digital and biotech, from outside the walls of leading industry players.

But incumbent corporations are only one part of the equation. The trillion dollars a year that companies spend on R&D is matched by the public sector. Well-funded start-ups, meanwhile, are developing and rapidly scaling innovations that often threaten to upset established business models or steer industry growth into new areas. Add increasing investor scrutiny of research spending, and the result is rising pressure on R&D leaders to quickly show results for their efforts.

R&D lacks connection to the customer. The R&D group tends to be isolated from the rest of the organization. The complexity of its activities and its specialized lexicon make it difficult for others to understand what the R&D function really does. That sense of working inside a “black box” often exists even within the R&D organization. During a meeting of one large company’s R&D leaders, a significant portion of the discussion focused on simply getting everyone up to speed on what the various divisions were doing, let alone connecting those efforts to the company’s broader goals.

Given the challenges R&D faces in collaborating with other functions, going one step further and connecting with customers becomes all the more difficult. While many organizations pay lip service to customer-centric development, their R&D groups rarely get the opportunity to test products directly with end users. This frequently results in market-back product development that relies on a game of telephone via many intermediaries about what the customers want and need.

Projects have few accountability metrics. R&D groups in most sectors lack effective mechanisms to measure and communicate progress; the pharmaceutical industry, with its standard pipeline for new therapeutics that provides well-understood metrics of progress and valuation implications, is the exception, not the rule. When failure is explained away as experimentation and success is described in terms of patents, rather than profits, corporate leaders find it hard to quantify R&D’s contribution.

Yet proven metrics exist  to effectively measure progress and outcomes. A common challenge we observe at R&D organizations, ranging from automotive to chemical companies, is how to value the contribution of a single component that is a building block of multiple products. One specialty-chemicals company faced this challenge in determining the value of an ingredient it used in its complex formulations. It created categorizations to help develop initial business cases and enable long-term tracking. This allowed pragmatic investment decisions at the start of projects and helped determine the value created after their completion.

Even with outcomes clearly measured, the often-lengthy period between initial investment and finished product can obscure the R&D organization’s performance. Yet, this too can be effectively managed by tracking the overall value and development progress of the pipeline so that the organization can react and, potentially, promptly reorient both the portfolio and individual projects within it.

Incremental projects get priority. Our research indicates that incremental projects account for more than half of an average company’s R&D investment, even though bold bets and aggressive reallocation  of the innovation portfolio deliver higher rates of success. Organizations tend to favor “safe” projects with near-term returns—such as those emerging out of customer requests—that in many cases do little more than maintain existing market share. One consumer-goods company, for example, divided the R&D budget among its business units, whose leaders then used the money to meet their short-term targets rather than the company’s longer-term differentiation and growth objectives.

Focusing innovation solely around the core business may enable a company to coast for a while—until the industry suddenly passes it by. A mindset that views risk as something to be avoided rather than managed can be unwittingly reinforced by how the business case is measured. Transformational projects at one company faced a higher internal-rate-of-return hurdle than incremental R&D, even after the probability of success had been factored into their valuation, reducing their chances of securing funding and tilting the pipeline toward initiatives close to the core.

As organizations mature, innovation-driven growth becomes increasingly important, as their traditional means of organic growth, such as geographic expansion and entry into untapped market segments, diminish. To succeed, they need to develop R&D strategies equipped for the modern era that treat R&D not as a cost center but as the growth engine it can become.

Would you like to learn more about our Strategy & Corporate Finance Practice ?

Choosing the ingredients of a winning r&d strategy.

Given R&D’s role as the innovation driver that advances the corporate agenda, its guiding strategy needs to link board-level priorities with the technologies that are the organization’s focus (Exhibit 2). The R&D strategy must provide clarity and commitment to three central elements: what we want to deliver, what we need to deliver it, and how we will deliver it.

What we want to deliver. To understand what a company wants to and can deliver, the R&D, commercial, and corporate-strategy functions need to collaborate closely, with commercial and corporate-strategy teams anchoring the R&D team on the company’s priorities and the R&D team revealing what is possible. The R&D strategy and the corporate strategy must be in sync while answering questions such as the following: At the highest level, what are the company’s goals? Which of these will require R&D in order to be realized? In short, what is the R&D organization’s purpose?

Bringing the two strategies into alignment is not as easy as it may seem. In some companies, what passes for corporate strategy is merely a five-year business plan. In others, the corporate strategy is detailed but covers only three to five years—too short a time horizon to guide R&D, especially in industries such as pharma or semiconductors where the product-development cycle is much longer than that. To get this first step right, corporate-strategy leaders should actively engage with R&D. That means providing clarity where it is lacking and incorporating R&D feedback that may illuminate opportunities, such as new technologies that unlock growth adjacencies for the company or enable completely new business models.

Secondly, the R&D and commercial functions need to align on core battlegrounds and solutions. Chief technology officers want to be close to and shape the market by delivering innovative solutions that define new levels of customer expectations. Aligning R&D strategy provides a powerful forum for identifying those opportunities by forcing conversations about customer needs and possible solutions that, in many companies, occur only rarely. Just as with the corporate strategy alignment, the commercial and R&D teams need to clearly articulate their aspirations by asking questions such as the following: Which markets will make or break us as a company? What does a winning product or service look like for customers?

When defining these essential battlegrounds, companies should not feel bound by conventional market definitions based on product groups, geographies, or customer segments. One agricultural player instead defined its markets by the challenges customers faced that its solutions could address. For example, drought resistance was a key battleground no matter where in the world it occurred. That framing clarified the R&D–commercial strategy link: if an R&D project could improve drought resistance, it was aligned to the strategy.

The dialogue between the R&D, commercial, and strategy functions cannot stop once the R&D strategy is set. Over time, leaders of all three groups should reexamine the strategic direction and continuously refine target product profiles as customer needs and the competitive landscape evolve.

What we need to deliver it. This part of the R&D strategy determines what capabilities and technologies the R&D organization must have in place to bring the desired solutions to market. The distinction between the two is subtle but important. Simply put, R&D capabilities are the technical abilities to discover, develop, or scale marketable solutions. Capabilities are unlocked by a combination of technologies and assets, and focus on the outcomes. Technologies, however, focus on the inputs—for example, CRISPR is a technology that enables the genome-editing capability.

This delineation protects against the common pitfall of the R&D organization fixating on components of a capability instead of the capability itself—potentially missing the fact that the capability itself has evolved. Consider the dawn of the digital age: in many engineering fields, a historical reliance on talent (human number crunchers) was suddenly replaced by the need for assets (computers). Those who focused on hiring the fastest mathematicians were soon overtaken by rivals who recognized the capability provided by emerging technologies.

The simplest way to identify the needed capabilities is to go through the development processes of priority solutions step by step—what will it take to produce a new product or feature? Being exhaustive is not the point; the goal is to identify high-priority capabilities, not to log standard operating procedures.

Prioritizing capabilities is a critical but often contentious aspect of developing an R&D strategy. For some capabilities, being good is sufficient. For others, being best in class is vital because it enables a faster path to market or the development of a better product than those of competitors. Take computer-aided design (CAD), which is used to design and prototype engineering components in numerous industries, such as aerospace or automotive. While companies in those sectors need that capability, it is unlikely that being the best at it will deliver a meaningful advantage. Furthermore, organizations should strive to anticipate which capabilities will be most important in the future, not what has mattered most to the business historically.

Once capabilities are prioritized, the R&D organization needs to define what being “good” and “the best” at them will mean over the course of the strategy. The bar rises rapidly in many fields. Between 2009 and 2019, the cost of sequencing a genome dropped 150-fold, for example. 2 Kris A. Wetterstrand, “DNA sequencing costs: Data,” NHGRI Genome Sequencing Program (GSP), August 25, 2020, genome.gov. Next, the organization needs to determine how to develop, acquire, or access the needed capabilities. The decision of whether to look internally or externally is crucial. An automatic “we can build it better” mindset diminishes the benefits of specialization and dilutes focus. Additionally, the bias to building everything in-house can cut off or delay access to the best the world has to offer—something that may be essential for high-priority capabilities. At Procter & Gamble, it famously took the clearly articulated aspiration of former CEO A. G. Lafley to break the company’s focus on in-house R&D and set targets for sourcing innovation externally. As R&D organizations increasingly source capabilities externally, finding partners and collaborating with them effectively is becoming a critical capability in its own right.

How we will do it. The choices of operating model and organizational design will ultimately determine how well the R&D strategy is executed. During the strategy’s development, however, the focus should be on enablers that represent cross-cutting skills and ways of working. A strategy for attracting, developing, and retaining talent is one common example.

Another is digital enablement, which today touches nearly every aspect of what the R&D function does. Artificial intelligence can be used at the discovery phase to identify emerging market needs or new uses of existing technology. Automation and advanced analytics approaches to experimentation can enable high throughput screening at a small scale and distinguish the signal from the noise. Digital (“in silico”) simulations are particularly valuable when physical experiments are expensive or dangerous. Collaboration tools are addressing the connectivity challenges common among geographically dispersed project teams. They have become indispensable in bringing together existing collaborators, but the next horizon is to generate the serendipity of chance encounters that are the hallmark of so many innovations.

Testing your R&D strategy

Developing a strategy for the R&D organization entails some unique challenges that other functions do not face. For one, scientists and engineers have to weigh considerations beyond their core expertise, such as customer, market, and economic factors. Stakeholders outside R&D labs, meanwhile, need to understand complex technologies and development processes and think along much longer time horizons than those to which they are accustomed.

For an R&D strategy to be robust and comprehensive enough to serve as a blueprint to guide the organization, it needs to involve stakeholders both inside and outside the R&D group, from leading scientists to chief commercial officers. What’s more, its definition of capabilities, technologies, talent, and assets should become progressively more granular as the strategy is brought to life at deeper levels of the R&D organization. So how can an organization tell if its new strategy passes muster? In our experience, McKinsey’s ten timeless tests of strategy  apply just as well to R&D strategy as to corporate and business-unit strategies. The following two tests are the most important in the R&D context:

• Does the organization’s strategy tap the true source of advantage? Too often, R&D organizations conflate technical necessity (what is needed to develop a solution) with strategic importance (distinctive capabilities that allow an organization to develop a meaningfully better solution than those of their competitors). It is also vital for organizations to regularly review their answers to this question, as capabilities that once provided differentiation can become commoditized and no longer serve as sources of advantage.
• Does the organization’s strategy balance commitment-rich choices with flexibility and learning? R&D strategies may have relatively long time horizons but that does not mean they should be insulated from changes in the outside world and never revisited. Companies should establish technical, regulatory, or other milestones that serve as clear decision points for shifting resources to or away from certain research areas. Such milestones can also help mark progress and gauge whether strategy execution is on track.

Additionally, the R&D strategy should be simply and clearly communicated to other functions within the company and to external stakeholders. To boost its clarity, organizations might try this exercise: distill the strategy into a set of fill-in-the-blank components that define, first, how the world will evolve and how the company plans to refocus accordingly (for example, industry trends that may lead the organization to pursue new target markets or segments); next, the choices the R&D function will make in order to support the company’s new focus (which capabilities will be prioritized and which de-emphasized); and finally, how the R&D team will execute the strategy in terms of concrete actions and milestones. If a company cannot fit the exercise on a single page, it has not sufficiently synthesized the strategy—as the famed physicist Richard Feynman observed, the ultimate test of comprehension is the ability to convey something to others in a simple manner.

Cascading the strategy down through the R&D organization will further reinforce its impact. For example, asking managers to communicate the strategy to their subordinates will deepen their own understanding. A useful corollary is that those hearing the strategy for the first time are introduced to it by their immediate supervisors rather than more distant R&D leaders. One R&D group demonstrated the broad benefits of this communication model: involving employees in developing and communicating the R&D strategy helped it double its Organizational Health Index  strategic clarity score, which measures one of the four “power practices”  highly connected to organizational performance.

R&D represents a massive innovation investment, but as companies confront globalized competition, rapidly changing customer needs, and technological shifts coming from an ever-wider range of fields, they are struggling to deliver on R&D’s full potential. A clearly articulated R&D strategy that supports and informs the corporate strategy is necessary to maximize the innovation investment and long-term company value.

Explore a career with us

Related articles.

The innovation commitment

The eight essentials of innovation

The Committed Innovator: An interview with Salesforce’s Simon Mulcahy

Research and Development (R&D)

• Reference work entry
• pp 5516–5517
• Cite this reference work entry

• Sakari Kainulainen 3  

544 Accesses

Mode 2 knowledge production (Mode 2) ; Mode 2 knowledge production ; Research and development and innovations (R&D&I)

Research and development (R&D) is a broad category describing the entity of basic research, applied research, and development activities. In general research and development means systematic activities in order to increase knowledge and use of this knowledge when developing new products, processes, or services. Nowadays innovation activities are strongly tight into the concept of research and development. In the broadest meaning, research and development consists of every activity from the basic research to the (successful) marketing of a product or (effective) launching of a new process (R&D&I).

Description

Research and development work is mostly related to business organizations. Development activities are targeted for them to development of new products and their success within the markets. A new product can be seen as the end of the chain of which...

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: The dynamics of science and research in contemporary societies . London: Sage.

Google Scholar  

Gulbrandsen, M. The role of basic research in innovation. Confluence , 55 . Retrieved from http://www.cas.uio.no/Publications/Seminar/Confluence_Gulbrandsen.pdf

Download references

Author information

Authors and affiliations.

Diaconia University of Applied Sciences, Sturenkatu 2, Helsinki, 00510, Finland

Sakari Kainulainen

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Sakari Kainulainen .

Editor information

Editors and affiliations.

University of Northern British Columbia, Prince George, BC, Canada

Alex C. Michalos

(residence), Brandon, MB, Canada

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer Science+Business Media Dordrecht

About this entry

Cite this entry.

Kainulainen, S. (2014). Research and Development (R&D). In: Michalos, A.C. (eds) Encyclopedia of Quality of Life and Well-Being Research. Springer, Dordrecht. https://doi.org/10.1007/978-94-007-0753-5_2482

Download citation

DOI : https://doi.org/10.1007/978-94-007-0753-5_2482

Publisher Name : Springer, Dordrecht

Print ISBN : 978-94-007-0752-8

Online ISBN : 978-94-007-0753-5

eBook Packages : Humanities, Social Sciences and Law

Share this entry

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

How Does Research and Development Influence Design?

How does research and development influence design? Research and development (R&D) is an integral part of any product design process. From concept to completion, R&D teams help bring ideas to life by testing the feasibility of new products and features.

In this blog post, we will explore how research can be used to inform decisions throughout a project’s lifecycle as well as discuss best practices for maximizing the impact that R&D has on design outcomes. We’ll also look at how technology can enhance traditional methods of conducting research, allowing teams to gain valuable insights faster than ever before. So let’s answer: how does research and development influence design?

Table of Contents

The Role of R&D in Design Processes

Market research, user testing, prototyping, leveraging technology to enhance research and development efforts, data management, collaboration, analytics and insights, challenges of leveraging technologies, best practices for maximizing the impact of research and development on design outcomes, conclusion: how does research and development influence design.

Research and development help to identify problems, develop solutions, and create new products or services that meet customer needs. R&D can also be used to improve existing designs by identifying areas for improvement or creating innovative approaches to problem-solving. Let’s look closer and answer: how does research and development influence design?

R&D plays a critical role in the design process by providing insights into customer needs and preferences, as well as technological advancements that could impact product performance.

Through research activities such as market analysis, surveys, prototype testing, and data collection from competitors’ products or services, designers gain valuable information about what their target audience wants and how best to deliver it. This knowledge can then be used to inform decisions about product features, materials selection, and manufacturing processes, resulting in improved designs that better meet user requirements.

Market research is a critical component of product development as it provides insights into consumer behavior and preferences. Through market research, designers can gain a better understanding of their target audience’s needs and wants which allows them to create more effective designs that appeal to customers.

For example, if a company wanted to launch a new line of clothing they could use market research data such as surveys or focus groups to determine what type of styles people prefer so they could tailor their designs accordingly.

User testing is another important aspect of product development as it allows designers to get feedback from real users on how well their products perform in practice. This information can be used by designers when making decisions about features or functionality so they can ensure that the result meets user expectations.

For instance, if an app was being developed, then user testing would help identify any potential usability issues before it was released so adjustments could be made accordingly.

Prototyping is also essential for successful product development as it allows designers to test out ideas before committing resources towards full-scale production. By creating prototypes early on in the process, designers can quickly iterate on concepts until they find one that works best for their intended purpose without having wasted time or money on something that may not have been viable in the long run anyway.

For example, if an automotive manufacturer wanted to develop a new car model then prototyping would allow them to experiment with different body shapes and materials. This will help them find one suitable for mass production at scale while minimizing costs associated with trial-and-error approaches.

Key Takeaway: R&D is an essential part of the design process, providing valuable insights into customer needs and technological advancements that can be used to inform decisions about product features.

Now that we’ve answered “how does research and development influence design,” let’s look at how to enhance R&D efforts. Leveraging technology for research and development (R&D) efforts can be a powerful tool to help teams achieve their goals. Technology can provide access to data, facilitate collaboration, and enable faster decision-making. Here are some of the benefits of leveraging technology for R&D efforts:

Technology provides access to large amounts of data that would otherwise be difficult or impossible to obtain. It also allows teams to collaborate more effectively by enabling them to share information quickly and easily across multiple locations. Additionally, technology enables faster decision-making by providing real-time insights into trends in the market or industry as well as competitor activities.

Organizing data is a key part of research and development. Leveraging technology can help streamline the process, making it easier for teams to access and analyze data quickly.

For example, Cypris provides an integrated platform that centralizes all the data sources R&D teams need into one place. This allows them to easily search through their information without having to switch between multiple systems or manually compile reports.

Technology also helps facilitate collaboration among team members who may be located in different parts of the world. By leveraging cloud-based tools such as Google Docs or Slack, researchers can work together on projects from anywhere with an internet connection.

These tools allow users to share documents, have conversations in real-time, assign tasks, and more – all within a single platform. Additionally, they provide version control so everyone is always working off the same document or set of instructions at any given time.

Finally, technology makes it easier for teams to uncover insights from their research by providing powerful analytics capabilities right out of the box. With the right analytics, teams can quickly identify trends in their data, make informed decisions about future projects, and develop new products faster than ever before.

That’s why R&D teams need to have a platform that provides comprehensive insights into their data.

One challenge is ensuring that the right technology is selected based on an organization’s specific needs and objectives. Another challenge is ensuring that the chosen technology integrates seamlessly with existing systems within an organization’s infrastructure so it can be utilized efficiently without disrupting operations or introducing security risks. Finally, there may also be challenges related to cost considerations when implementing new technologies such as software licensing fees or hardware costs associated with deploying new systems or upgrading existing ones.

Key Takeaway: Technology can be a powerful tool for R&D teams to help them achieve their goals by providing access to data, facilitating collaboration, and enabling faster decision-making. However, organizations must consider cost considerations when selecting the right technology that integrates seamlessly with existing systems without introducing security risks.

Research and development (R&D) is an essential component of any successful design process. To maximize the impact of R&D on design outcomes, teams should focus on integrating research into their processes early and often.

This includes setting up a feedback loop between research and design to ensure that insights from research are informing decisions throughout the entire process. Additionally, teams should strive to create a culture where experimentation is encouraged, as this will allow them to explore different solutions quickly and efficiently.

Apple is one company that has successfully leveraged best practices for maximizing the impact of R&D on design outcomes. By creating a strong feedback loop between their research team and product designers, they have been able to rapidly develop innovative products such as iPhones and iPads.

Similarly, Amazon has also used its in-house research team to inform its product designs; by leveraging customer data collected through its platform, Amazon has been able to create highly personalized experiences tailored specifically to each user’s needs.

One challenge with implementing best practices for maximizing the impact of R&D on design outcomes is finding ways to effectively communicate insights from research back into product development cycles without sacrificing speed or efficiency. Additionally, it can be difficult to find ways to incentivize collaboration between researchers and designers so that both groups are working together towards common goals instead of operating independently from one another.

Finally, there may be organizational challenges associated with establishing an effective feedback loop between these two groups if they exist within separate departments or silos within an organization’s structure.

Key Takeaway: To maximize the impact of R&D on design outcomes, teams should focus on creating a feedback loop between research and design that encourages experimentation. Challenges may arise from communication issues or organizational silos, but with proper planning. these can be overcome.

How does research and development influence design? Research and development is an essential part of the design process, as it provides valuable insight into customer needs and preferences which can be used to inform decision-making throughout the entire product lifecycle.

By leveraging technology to enhance R&D efforts, teams can maximize their impact on product innovation and ensure they are making informed decisions based on data-driven insights. Ultimately, understanding how research and development influence design is key for any organization looking to stay ahead of the competition in today’s ever-evolving market landscape.

Are you an R&D or innovation team looking for a platform to accelerate your time to insights? Cypris is the perfect solution. Our research platform has been specifically designed with teams in mind and provides easy access to data sources that can help take your projects from concept to completion quickly. Take advantage of our innovative technology today and see how much faster your ideas become reality!

Similar insights you might enjoy

Gallium Nitride Innovation Pulse

Carbon Capture & Storage Innovation Pulse

Featured in:

Companies often spend resources on certain investigative undertakings in an effort to make discoveries that can help develop new products or way of doing things or work towards enhancing pre-existing products or processes. These activities come under the Research and Development (R&D) umbrella.

R&D is an important means for achieving future growth and maintaining a relevant product in the market . There is a misconception that R&D is the domain of high tech technology firms or the big pharmaceutical companies. In fact, most established consumer goods companies dedicate a significant part of their resources towards developing new versions of products or improving existing designs . However, where most other firms may only spend less than 5 percent of their revenue on research, industries such as pharmaceutical, software or high technology products need to spend significantly given the nature of their products.

© Shutterstock.com | Alexander Raths

In this article, we look at 1) types of R&D , 2) understanding similar terminology , 3) making the R&D decision , 4) basic R&D process , 5) creating an effective R&D process , 6) advantages of R&D , and 7) R&D challenges .

TYPES OF R&D

A US government agency, the National Science Foundation defines three types of R&D .

Basic Research

When research aims to understand a subject matter more completely and build on the body of knowledge relating to it, then it falls in the basic research category. This research does not have much practical or commercial application. The findings of such research may often be of potential interest to a company

Applied Research

Applied research has more specific and directed objectives. This type of research aims to determine methods to address a specific customer/industry need or requirement. These investigations are all focused on specific commercial objectives regarding products or processes.

Development

Development is when findings of a research are utilized for the production of specific products including materials, systems and methods. Design and development of prototypes and processes are also part of this area. A vital differentiation at this point is between development and engineering or manufacturing. Development is research that generates requisite knowledge and designs for production and converts these into prototypes. Engineering is utilization of these plans and research to produce commercial products.

UNDERSTANDING SIMILAR TERMINOLOGY

There are a number of terms that are often used interchangeably. Thought there is often overlap in all of these processes, there still remains a considerable difference in what they represent. This is why it is important to understand these differences.

The creation of new body of knowledge about existing products or processes, or the creation of an entirely new product is called R&D. This is systematic creative work, and the resulting new knowledge is then used to formulate new materials or entire new products as well as to alter and improve existing ones

Innovation includes either of two events or a combination of both of them. These are either the exploitation of a new market opportunity or the development and subsequent marketing of a technical invention. A technical invention with no demand will not be an innovation.

New Product Development

This is a management or business term where there is some change in the appearance, materials or marketing of a product but no new invention. It is basically the conversion of a market need or opportunity into a new product or a product upgrade

When an idea is turned into information which can lead to a new product then it is called design. This term is interpreted differently from country to country and varies between analytical marketing approaches to a more creative process.

Product Design

Misleadingly thought of as the superficial appearance of a product, product design actually encompasses a lot more. It is a cross functional process that includes market research, technical research, design of a concept, prototype creation, final product creation and launch . Usually, this is the refinement of an existing product rather than a new product.

MAKING THE R&D DECISION

Investment in R&D can be extensive and a long term commitment. Often, the required knowledge already exists and can be acquired for a price. Before committing to investment in R&D, a company needs to analyze whether it makes more sense to produce their own knowledge base or acquire existing work. The influence of the following factors can help make this decision.

Proprietariness

If the nature of the research is such that it can be protected through patents or non-disclosure agreements , then this research becomes the sole property of the company undertaking it and becomes much more valuable. Patents can allow a company several years of a head start to maximize profits and cement its position in the market. This sort of situation justifies the cost of the R&D process. On the other hand, if the research cannot be protected, then it may be easily copied by a competitor with little or no monetary expense. In this case, it may be a good idea to acquire research.

Setting up a R&D wing only makes sense if the market growth rate is slow or relatively moderate. In a fast paced environment, competitors may rush ahead before research has been completed, making the entire process useless.

Because of its nature, R&D is not always a guaranteed success commercially. In this regard, it may be desirable to acquire the required research to convert it into necessary marketable products. There is significantly less risk in acquisition as there may be an opportunity to test the technology out before formally purchasing anything.

Considering the long term potential success of a product, acquiring technology is less risky but more costly than generating own research. This is because license fees or royalties may need to be paid and there may even be an arrangement that requires payments tied to sales figures and may continue for as long as the license period. There is also the danger of geographical limitations or other restrictive caveats. In addition, if the technology changes mid license, all the investment will become a sunk cost. Setting up R&D has its own costs associated with it. There needs to be massive initial investment that leads to negative cash flow for a long time. But it does protect the company from the rest of the limitations of acquiring research.

All these aspects need to be carefully assessed and a pros vs. cons assessment needs to be conducted before the make or buy decision is finalized.

BASIC R&D PROCESS

Foster Ideas

At this point the research team may sit down to brainstorm. The discussion may start with an understanding and itemization of the issues faced in their particular industry and then narrowed down to important or core areas of opportunity or concern.

Focus Ideas

The initial pool of ideas is vast and may be generic. The team will then sift through these and locate ideas with potential or those that do not have insurmountable limitations. At this point the team may look into existing products and assess how original a new idea is and how well it can be developed.

Develop Ideas

Once an idea has been thoroughly researched, it may be combined with a market survey to assess market readiness. Ideas with true potential are once again narrowed down and the process of turning research into a marketable commodity begins.

Prototypes and Trials

Researchers may work closely with product developers to understand and agree on how an idea may be turned into a practical product. As the process iterates, the prototype complexity may start to increase and issues such as mass production and sales tactics may begin to enter the process.

Regulatory, Marketing & Product Development Activities

As the product takes shape, the process that began with R&D divides into relevant areas necessary to bring the research product to the market. Regulatory aspects are assessed and work begins to meet all the criteria for approvals and launch. The marketing function begins developing strategies and preparing their materials while sales, pricing and distribution are also planned for.

The product that started as a research question will now be ready for its biggest test, the introduction to the market. The evaluation of the product continues at this stage and beyond, eventually leading to possible re-designs if needed. At any point in this process the idea may be abandoned. Its feasibility may be questioned or the research may not reveal what the business hoped for. It is therefore important to analyze each idea critically at every stage and not become emotionally invested in anything.

CREATING AN EFFECTIVE R&D PROCESS

A formal R&D function adds great value to any organization. It can significantly contribute towards organizational growth and sustained market share. However, all business may not have the necessary resources to set up such a function. In such cases, or in organizations where a formal R&D function is not really required, it is a good idea to foster an R&D mindset . When all employees are encouraged to think creatively and with a research oriented thought process, they all feel invested in the business and there will be the possibility of innovation and unique ideas and solutions. This mindset can be slowly inculcated within the company by following the steps mentioned below.

Assess Customer Needs

It is a good idea to regularly scan and assess the market and identify whether the company’s offering is doing well or if it is in trouble. If it is successful, encourage employees to identify reasons for success so that these can then be used as benchmarks or best practices. If the product is not doing well, then encourage teams to research reasons why. Perhaps a competitor is offering a better solution or perhaps the product cannot meet the customer’s needs effectively.

Identify Objectives

Allow your employees to see clearly what the business objectives are. The end goal for a commercial enterprise is to enhance profits. If this is the case, then all research the employees engage in should focus on reaching this goal while fulfilling a customer need.

Define and Design Processes

A definite project management process helps keep formal and informal research programs on schedule. Realistic goals and targets help focus the process and ensures that relevant and realistic timelines are decided upon.

Create a Team

A team may need to be created if a specific project is on the agenda. This team should be cross functional and will be able to work towards a specific goal in a systematic manner. If the surrounding organizational environment also has a research mindset then they will be better prepared and suited to assist the core team when ever needed.

Whenever needed, it may be a good idea to outsource research projects. Universities and specific research organizations can help achieve research objectives that may not be manageable within a limited organizational budget.

ADVANTAGES OF R&D

Though setting up an R&D function is not an easy task by any means, it has its unique advantages for the organization. These include the following.

Research and Development expenses are often tax deductible. This depends on the country of operations of course but a significant write-off can be a great way to offset large initial investments. But it is important to understand what kind of research activities are deductible and which ones are not. Generally, things like market research or an assessment of historical information are not deductible.

A company can use research to identify leaner and more cost effective means of manufacturing. This reduction in cost can either help provide a more reasonably priced product to the customer or increase the profit margin.

When an investor sets out to put their resources into any company, they tend to prefer those who can become market leaders and innovate constantly. An effective R&D function goes a long way in helping to achieve these objectives for a company. Investors see this as a proactive approach to business and they may end up financing the costs associated with maintaining this R&D function.

Recruitment

Top talent is also attracted to innovative companies doing exciting things. With a successful Research and Development function, qualified candidates will be excited to join the company.

Through R&D based developments, companies can acquire patents for their products. These can help them gain market advantage and cement their position in the industry. This one time product development can lead to long term profits.

R&D CHALLENGES

R&D also has many challenges associated with it. These may include the following.

Initial setup costs as well as continued investment are necessary to keep research work cutting edge and relevant. Not all companies may find it feasible to continue this expenditure.

Increased Timescales

Once a commitment to R&D is made, it may take many years for the actual product to reach the market and a number of years will be filled with no return on continued heavy investment.

Uncertain Results

Not all research that is undertaken yields results. Many ideas and solutions are scrapped midway and work has to start from the beginning.

Market Conditions

There is always the danger that a significant new invention or innovation will render years of research obsolete and create setbacks in the industry with competitors becoming front runners for the customer’s business.

It is important for any business to understand the advantages and disadvantages of engaging in Research and Development activities. Once these are studied, then the step can be taken towards becoming and R&D organization.

In the meanwhile, it is good practice to inculcate a research mind set and research oriented thinking within all employees, no matter what their functional area of expertise. This will help bring about new ideas, new solutions and an innovative way of approaching all business problems, whether small or large.

Comments are closed.

Related posts

6 Steps Help You Coach Employees Effectively

Most employers do not always review the performance of their employees except the traditional annual …

Data Storytelling Links Emotions and Rational Decisions

Did you know the human brain is more likely to take in information from a story over pure …

Mark Cuban’s 12 Rules for Startups

Mark Cuban is a billionaire investor and owner of the Dallas Mavericks, with a lot of experience …

408,000 + job opportunities

Not yet a member? Sign Up

join cleverism

Find your dream job. Get on promotion fasstrack and increase tour lifetime salary.

Post your jobs & get access to millions of ambitious, well-educated talents that are going the extra mile.

First name*

Company name*

Company Website*

E-mail (work)*

Login or Register

Password reset instructions will be sent to your E-mail.

Module 1: Lifespan Development

Developmental research designs, learning outcomes.

• Compare advantages and disadvantages of developmental research designs (cross-sectional, longitudinal, and sequential)

Now you know about some tools used to conduct research about human development. Remember,  research methods  are tools that are used to collect information. But it is easy to confuse research methods and research design. Research design is the strategy or blueprint for deciding how to collect and analyze information. Research design dictates which methods are used and how. Developmental research designs are techniques used particularly in lifespan development research. When we are trying to describe development and change, the research designs become especially important because we are interested in what changes and what stays the same with age. These techniques try to examine how age, cohort, gender, and social class impact development.

Cross-sectional designs

The majority of developmental studies use cross-sectional designs because they are less time-consuming and less expensive than other developmental designs. Cross-sectional research designs are used to examine behavior in participants of different ages who are tested at the same point in time. Let’s suppose that researchers are interested in the relationship between intelligence and aging. They might have a hypothesis (an educated guess, based on theory or observations) that intelligence declines as people get older. The researchers might choose to give a certain intelligence test to individuals who are 20 years old, individuals who are 50 years old, and individuals who are 80 years old at the same time and compare the data from each age group. This research is cross-sectional in design because the researchers plan to examine the intelligence scores of individuals of different ages within the same study at the same time; they are taking a “cross-section” of people at one point in time. Let’s say that the comparisons find that the 80-year-old adults score lower on the intelligence test than the 50-year-old adults, and the 50-year-old adults score lower on the intelligence test than the 20-year-old adults. Based on these data, the researchers might conclude that individuals become less intelligent as they get older. Would that be a valid (accurate) interpretation of the results?

Figure 1 . Example of cross-sectional research design

No, that would not be a valid conclusion because the researchers did not follow individuals as they aged from 20 to 50 to 80 years old. One of the primary limitations of cross-sectional research is that the results yield information about age differences  not necessarily changes with age or over time. That is, although the study described above can show that in 2010, the 80-year-olds scored lower on the intelligence test than the 50-year-olds, and the 50-year-olds scored lower on the intelligence test than the 20-year-olds, the data used to come up with this conclusion were collected from different individuals (or groups of individuals). It could be, for instance, that when these 20-year-olds get older (50 and eventually 80), they will still score just as high on the intelligence test as they did at age 20. In a similar way, maybe the 80-year-olds would have scored relatively low on the intelligence test even at ages 50 and 20; the researchers don’t know for certain because they did not follow the same individuals as they got older.

It is also possible that the differences found between the age groups are not due to age, per se, but due to cohort effects. The 80-year-olds in this 2010 research grew up during a particular time and experienced certain events as a group. They were born in 1930 and are part of the Traditional or Silent Generation. The 50-year-olds were born in 1960 and are members of the Baby Boomer cohort. The 20-year-olds were born in 1990 and are part of the Millennial or Gen Y Generation. What kinds of things did each of these cohorts experience that the others did not experience or at least not in the same ways?

You may have come up with many differences between these cohorts’ experiences, such as living through certain wars, political and social movements, economic conditions, advances in technology, changes in health and nutrition standards, etc. There may be particular cohort differences that could especially influence their performance on intelligence tests, such as education level and use of computers. That is, many of those born in 1930 probably did not complete high school; those born in 1960 may have high school degrees, on average, but the majority did not attain college degrees; the young adults are probably current college students. And this is not even considering additional factors such as gender, race, or socioeconomic status. The young adults are used to taking tests on computers, but the members of the other two cohorts did not grow up with computers and may not be as comfortable if the intelligence test is administered on computers. These factors could have been a factor in the research results.

Another disadvantage of cross-sectional research is that it is limited to one time of measurement. Data are collected at one point in time and it’s possible that something could have happened in that year in history that affected all of the participants, although possibly each cohort may have been affected differently. Just think about the mindsets of participants in research that was conducted in the United States right after the terrorist attacks on September 11, 2001.

Longitudinal research designs

Figure 2 . Longitudinal research studies the same person or group of people over an extended period of time.

Longitudinal   research involves beginning with a group of people who may be of the same age and background (cohort) and measuring them repeatedly over a long period of time. One of the benefits of this type of research is that people can be followed through time and be compared with themselves when they were younger; therefore changes with age over time are measured. What would be the advantages and disadvantages of longitudinal research? Problems with this type of research include being expensive, taking a long time, and participants dropping out over time. Think about the film, 63 Up , part of the Up Series mentioned earlier, which is an example of following individuals over time. In the videos, filmed every seven years, you see how people change physically, emotionally, and socially through time; and some remain the same in certain ways, too. But many of the participants really disliked being part of the project and repeatedly threatened to quit; one disappeared for several years; another died before her 63rd year. Would you want to be interviewed every seven years? Would you want to have it made public for all to watch?   

Longitudinal research designs are used to examine behavior in the same individuals over time. For instance, with our example of studying intelligence and aging, a researcher might conduct a longitudinal study to examine whether 20-year-olds become less intelligent with age over time. To this end, a researcher might give an intelligence test to individuals when they are 20 years old, again when they are 50 years old, and then again when they are 80 years old. This study is longitudinal in nature because the researcher plans to study the same individuals as they age. Based on these data, the pattern of intelligence and age might look different than from the cross-sectional research; it might be found that participants’ intelligence scores are higher at age 50 than at age 20 and then remain stable or decline a little by age 80. How can that be when cross-sectional research revealed declines in intelligence with age?

Figure 3 . Example of a longitudinal research design

Since longitudinal research happens over a period of time (which could be short term, as in months, but is often longer, as in years), there is a risk of attrition. Attrition occurs when participants fail to complete all portions of a study. Participants may move, change their phone numbers, die, or simply become disinterested in participating over time. Researchers should account for the possibility of attrition by enrolling a larger sample into their study initially, as some participants will likely drop out over time. There is also something known as  selective attrition— this means that certain groups of individuals may tend to drop out. It is often the least healthy, least educated, and lower socioeconomic participants who tend to drop out over time. That means that the remaining participants may no longer be representative of the whole population, as they are, in general, healthier, better educated, and have more money. This could be a factor in why our hypothetical research found a more optimistic picture of intelligence and aging as the years went by. What can researchers do about selective attrition? At each time of testing, they could randomly recruit more participants from the same cohort as the original members, to replace those who have dropped out.

The results from longitudinal studies may also be impacted by repeated assessments. Consider how well you would do on a math test if you were given the exact same exam every day for a week. Your performance would likely improve over time, not necessarily because you developed better math abilities, but because you were continuously practicing the same math problems. This phenomenon is known as a practice effect. Practice effects occur when participants become better at a task over time because they have done it again and again (not due to natural psychological development). So our participants may have become familiar with the intelligence test each time (and with the computerized testing administration).

Another limitation of longitudinal research is that the data are limited to only one cohort. As an example, think about how comfortable the participants in the 2010 cohort of 20-year-olds are with computers. Since only one cohort is being studied, there is no way to know if findings would be different from other cohorts. In addition, changes that are found as individuals age over time could be due to age or to time of measurement effects. That is, the participants are tested at different periods in history, so the variables of age and time of measurement could be confounded (mixed up). For example, what if there is a major shift in workplace training and education between 2020 and 2040 and many of the participants experience a lot more formal education in adulthood, which positively impacts their intelligence scores in 2040? Researchers wouldn’t know if the intelligence scores increased due to growing older or due to a more educated workforce over time between measurements.

Sequential research designs

Sequential research designs include elements of both longitudinal and cross-sectional research designs. Similar to longitudinal designs, sequential research features participants who are followed over time; similar to cross-sectional designs, sequential research includes participants of different ages. This research design is also distinct from those that have been discussed previously in that individuals of different ages are enrolled into a study at various points in time to examine age-related changes, development within the same individuals as they age, and to account for the possibility of cohort and/or time of measurement effects. In 1965, K. Warner Schaie [1] (a leading theorist and researcher on intelligence and aging), described particular sequential designs: cross-sequential, cohort sequential, and time-sequential. The differences between them depended on which variables were focused on for analyses of the data (data could be viewed in terms of multiple cross-sectional designs or multiple longitudinal designs or multiple cohort designs). Ideally, by comparing results from the different types of analyses, the effects of age, cohort, and time in history could be separated out.

Consider, once again, our example of intelligence and aging. In a study with a sequential design, a researcher might recruit three separate groups of participants (Groups A, B, and C). Group A would be recruited when they are 20 years old in 2010 and would be tested again when they are 50 and 80 years old in 2040 and 2070, respectively (similar in design to the longitudinal study described previously). Group B would be recruited when they are 20 years old in 2040 and would be tested again when they are 50 years old in 2070. Group C would be recruited when they are 20 years old in 2070 and so on.

Figure 4. Example of sequential research design

Studies with sequential designs are powerful because they allow for both longitudinal and cross-sectional comparisons—changes and/or stability with age over time can be measured and compared with differences between age and cohort groups. This research design also allows for the examination of cohort and time of measurement effects. For example, the researcher could examine the intelligence scores of 20-year-olds in different times in history and different cohorts (follow the yellow diagonal lines in figure 3). This might be examined by researchers who are interested in sociocultural and historical changes (because we know that lifespan development is multidisciplinary). One way of looking at the usefulness of the various developmental research designs was described by Schaie and Baltes (1975) [2] : cross-sectional and longitudinal designs might reveal change patterns while sequential designs might identify developmental origins for the observed change patterns.

Since they include elements of longitudinal and cross-sectional designs, sequential research has many of the same strengths and limitations as these other approaches. For example, sequential work may require less time and effort than longitudinal research (if data are collected more frequently than over the 30-year spans in our example) but more time and effort than cross-sectional research. Although practice effects may be an issue if participants are asked to complete the same tasks or assessments over time, attrition may be less problematic than what is commonly experienced in longitudinal research since participants may not have to remain involved in the study for such a long period of time.

When considering the best research design to use in their research, scientists think about their main research question and the best way to come up with an answer. A table of advantages and disadvantages for each of the described research designs is provided here to help you as you consider what sorts of studies would be best conducted using each of these different approaches.

• Schaie, K.W. (1965). A general model for the study of developmental problems. Psychological Bulletin, 64(2), 92-107. ↵
• Schaie, K.W. & Baltes, B.P. (1975). On sequential strategies in developmental research: Description or Explanation. Human Development, 18: 384-390. ↵
• Modification, adaptation, and original content. Authored by : Margaret Clark-Plaskie for Lumen Learning. Provided by : Lumen Learning. License : CC BY-NC-SA: Attribution-NonCommercial-ShareAlike
• Research Methods in Developmental Psychology. Authored by : Angela Lukowski and Helen Milojevich. Provided by : University of Calfornia, Irvine. Located at : https://nobaproject.com/modules/research-methods-in-developmental-psychology?r=LDcyNTg0 . Project : The Noba Project. License : CC BY-NC-SA: Attribution-NonCommercial-ShareAlike
• Woman holding own photograph. Provided by : Pxhere. Located at : https://pxhere.com/en/photo/221167 . License : CC0: No Rights Reserved

Ohio State nav bar

Ohio state navigation bar.

• BuckeyeLink
• Search Ohio State

MFA: Design Research and Development (DRD)

The Design Research and Development track is for persons with academic backgrounds in all fields of design. Professional practice experience is also useful, but is not required. Students in our program are encouraged to think critically about design problems and processes, to compare and test methods, and to seek discoveries through the analysis and synthesis of cross-disciplinary approaches and the application of design thinking. Because the program is intentionally multidisciplinary and process-oriented, courses in this degree may also be of interest to persons who are incorporating design thinking into the processes of other disciplines, which generates a diverse atmosphere. Research topics are encouraged in alignment with the expertise of the core faculty in order to maximize the potential for student participation in funded projects and other active theoretical or practical investigations. Areas of investigation generally fall into one of three categories: Research for design practices; Research about design practices; Design practices as research.

Students in this track are required to complete 60 credit hours of study to receive a MFA degree in Design. The program takes three years to complete, and only under exceptional circumstances is the track completed in less time. The curriculum includes a core of seminar and studio courses and independent thesis-related studies and open electives in the themes of studio/lab, history/theory, and collaborative/interdisciplinary activity. Students work closely with a three-person thesis committee to develop their thesis topics and the vehicles best used for their development. 

The Design Research and Development (DRD) track, leading to the Master of Fine Arts (MFA) in Design, is devoted to the development of knowledge and skills possessed by successful designers, including environmental and social awareness, novel methods for human-centered and systematic analysis and visualization, and creative problem-solving. We emphasize the creation of outcomes that address and contribute to contemporary issues in a global society.

The DRD track centers on research that discovers and explores new concepts, theories and outcomes in domains that are relevant to designers. The techniques and tools applied may be those traditional to design practice, including testing performance, usability, and user satisfaction, or they may evolve from experimentation, prototyping/making and exploration of the digital realm. Engaging primary research is critical to the processes of discovery that we encourage.

Example Course Plan

• Second Year

Autumn Semester

DESIGN 6100 Methods for Design Research, 3 credits DESIGN 6200 Foundation for Grad Studies, 3 credits DESIGN 6400 Graduate Studio, 3 credits

Spring Semester

DESIGN 6300 Exploration in Graduate Design Studies, 3 credits DESIGN 6400 Graduate Studio, 3 credits DESIGN 7193 with 2-mentors, Independent Study, 3 credits  Program Milestone: First Year Review (Research Orientation)

Electives, 4-6 credits 

• For Autumn and Spring semesters, students on 50% Graduate Associateship appointments must be enrolled for a minimum of 8 credits. For a 25% appointment, the minimum is 4 credits. 
• Students awarded a Fellowship must enroll for a minimum of 12 credits for Autumn and Spring semester, and at least 6 credits for Summer semester.
• Internship (up to 6 credits) can fulfill Studio Elective or Interdisciplinary Elective credits depending on the nature of the work experience. This is determined by the advisor.
• Elective courses maybe taken in any order, and within Design or in other departments or centers at OSU.

DESIGN 6400 Graduate Studio, 3 credits DESIGN 6500 Graduate Design Seminar, 3 credits Elective (AAEP 7725 Research Writing recommended), 3 credits 

DESIGN 6400 Design Graduate Studio, 3 credits DESIGN 7998 Thesis Project Development, 3 credits Elective, 3 credits 

Program Milestone: Second Year Review (Research Proposal & Plan)

Elective, 4-6 credits 

• For Autumn and Spring semesters, students on 50% Graduate Associateship appointments must be enrolled for a minimum of 8 credits. For a 25% appointment, the minimum is 4 credits.
• Elective courses maybe taken in any order, and within Design or in other departments or centers at OSU.  

DESIGN 7998 Thesis Project Development, 3 credits  DESIGN 7999 Thesis Writing, 3 credits  Elective, 3 credits 

DESIGN 7998 Thesis Project Development, 3 credits DESIGN 7999 Thesis Writing, 3 credits  Elective, 3 credits  Thesis Defense: Project Exhibition, Oral Defense

• Students awarded a Fellowship must enroll for a minimum of 12 credits for Autumn and Spring semester, and at least 6 credits for Summer semester.  

Curriculum Credit Distribution

Courses include: DESIGN 6100, DESIGN 6200, DESIGN 6300, DESIGN 6400, DESIGN 6500, DESIGN 7998, and DESIGN 7999. 

Individual Studies (DESIGN 7193) that explore a research interest in design under the guidance of design faculty.

Courses providing hands-on experiences that advance skill applied in creative research or to understand and incorporate experimental or applied practices in any discipline (i.e. ACCAD Grad Studio Courses - 5001, 5100, 5140, 5301, 6002, 7103).

Courses providing strong emphasis on history and analytical methods, as well as the philosophical, political, material, and context for works related to design, animation, or interactive media and specific research interests.

Courses providing opportunities to study and interact with another discipline beyond design.

For detailed information about forming Thesis Committees, the first year review and Thesis Committee Review, please see the Department of Design Graduate Studies Handbook .

Career Prospects

• Brand Designer/Manager
• Data/Science Visualizer
• Design Director/Manager
• Design Educator
• Design Researcher
• Design Strategist
• Graphic Designer
• Human-Computer Interaction Designer
• Industrial Designer
• Interaction Designer
• Interior Designer
• Motion Graphic Designer
• Multimedia Designer
• Product Designer
• User Experience (UX) Designer
• User Interface (UI) Designer
• User Interface (UI) Research
• User Experience (UX) Strategist
• User Experience (UX) Researcher
• Visual Communication Designer
• Web and Social Media Designer/Manager

Research Gallery: Example DRD Projects 

See our Research Gallery or click the links below to view past examples of DRD projects featured at our annual Graduate Spring Exhibition, which occurs at Hopkins Hall Gallery. 

Designing for Breast Cancer Survivors’ Empowerment: Integration of Technology for Self-management Promotion through Participatory Design

Sana Behnam

MFA Design Research and Development, 2021

Facilitating a Creative Growth Mindset: A Creative Process that Integrates Gameplay with Maketools

MFA Design Research and Development, 2020

Prototyping with Co-designers to Imagine Future Experiences

David McKenzie

MFA Design Research and Development, 2016

• Section 2: Home
• Developing the Quantitative Research Design
• Qualitative Descriptive Design
• Design and Development Research (DDR) For Instructional Design

Overview of Design and Development Research (DDR) For Applied Doctorate Students in the Instructional Design Program

Types of design and development research, 3 stages in design and development research, data collection methods and sources of data in ddr.

• Qualitative Narrative Inquiry Research
• Action Research Resource
• Case Study Design in an Applied Doctorate
• SAGE Research Methods
• Research Examples (SAGE) This link opens in a new window
• Dataset Examples (SAGE) This link opens in a new window
• IRB Resource Center This link opens in a new window

The purpose of this quick guide is to assist Applied Doctorate students in the Instructional Design Program in determining the best methodology and design for their Applied Doctorate Experience (ADE) dissertation. The guide covers intended target audience, an overview of Design and Development Research (DDR), types of DDR research including product, program, tool research, and model research, 3 stages providing alignment of DDR with NUs Applied Doctoral Record (DDR) deliverables, examples of problem, purpose, and research questions for DDR research, and suggested references. 

Target Audience: Doctoral Students in Instructional Design in the ADE program

This quick reference guide will aid doctoral students in instructional design challenged with deciding on what type of applied research study they want to do for their dissertation.

Overview of Design and Development Research (DDR) Methods

At the core of the instructional design and instructional technology and media field, is the design, development, implementation and evaluation of instructional products, tools, programs, models, and frameworks.  In many ways DDR is like Action Research (Goldkuhl, 2012), however, there are many differences. DDR research allows instructional designers a pathway to test theory, models, and frameworks and to authenticate practice. The focus of DDR is on the use, design, development, implementation, and evaluation of products, tools, programs, and models using instructional design models and frameworks. Richey and Klein (2007) defined DDR as “the systematic study of design, development, and evaluation processes with the aim of establishing an empirical basis for the creation of instructional and non-instructional products and tools and new or enhanced models that govern their development” (p. xv). Often the models and frameworks are validated and/or further developed and enhanced through the DDR. DDR is applied research. An area of DDR research that is particularly applicable to ADE students is the creation, implementation, and evaluation of one or more artifacts, such as products, tools, models, new technologies, and learning objects that will aid in solving a complex problem in practice that can be addressed through human imagination, creativity, engagement, and interaction (Ellis & Levy, 2010). These types of problems are found in K-12 education, higher education, corporations, not-for-profits, healthcare, and the military. 

• Example Design and Development Research

The field of DDR is constantly evolving and expanding as technology and media are changing at exponential rates.  Richey and Klein (2007) in their seminal work divided DDR into two major categories:

• Product and Tool Research and
• Model Research.

Table 1 provides a summary of common designs used in DDR. Most DDR work falls under the qualitative research category of qualitative case study, however, methodologies such as quantitative and mixed method have been used as well as other qualitative designs, including Delphi.

Product, Program, and Tool Research

Ellis and Levy (2010) asserted that DDR must go beyond commercial product development by determining a research problem, based on existing research literature and gaps in the literature that researchers assert must be studied to add to the instructional design knowledgebase.

Product and Tool Research can be further divided into:

• Comprehensive Design and Development Projects covering all phases of the instructional design process,
• Specific Project Phases (such as those in the ADDIE model: Analysis, Design, Development, Implementation, and Evaluation), and
• Design, Development, and Use of tools (Richey & Klein, 2007).

Model Research

Instructional designers and instructional technologists have focused on model research since the emergence of the field.

Model research can be broken into three types:

• Model Development,
• Model Validation and

Model development can focus on a comprehensive model design or on part of a process. Model validation research uses empirical processes to prove the effectiveness of a model in practice. Finally, model use research addresses usability typically from the perspective of instructional designers and stakeholder experts.

3 Stages in Design and Development Research for the ADE Doctoral Student’s Dissertation

NU doctoral students in the Instructional Design Program can use one of the various types of DDR research to complete their doctoral dissertation using the NU ADE template. There will be three stages in this process and in each stage the student will have one or more deliverables using the NU template and posting in the ADR on NU One.

Stage 1: Design and Development Research aligned with the NU ADE Template Process

• Identify a research worthy problem which is expressed by researchers in peer reviewed research literature. Ask yourself, what is going wrong? What do researchers say is known about the problem? And what is needed to be known to address the problem?
• Describe the purpose of your research ensuring that it aligns with your problem statement. In the description state your methodology and design and which DDR type of research you will do. Be sure to include a description of your target population (audience), the size of your sample and the sampling strategy you will use to access your sample. What permissions do you need? Site permission? Other IRB permission?
• Write your research questions to align with your problem and purpose statements.
• Complete Section 1 of your Applied Doctoral Experience (ADE) template securing all necessary approvals in the Applied Doctoral Record (ADR).
• Needs Assessment
• Measurable Goals and Objectives
• Sample size and Access to the sample
• Sampling strategy
• Content analysis (course, program, product, or tool descriptions)
• Technology and media analysis/selection
• Learning management system(s)
• Asynchronous
• Synchronous
• Evaluation Plan
• Complete Section 2, Proposal Draft, Proposal for AR, and Final Proposal of the ADE securing all necessary approvals in the ADR.
• Submit Proposal and IRB Application to secure IRB approval.

Stage 2: Design and Development Research aligned with the NU ADE Template Process

After receiving IRB approval of your ADE Proposal, it is time to design, develop, test, validate, and/or evaluate your artifacts. Below are example steps:

• Review and Finalize Design Document
• Recruit Expert Participants, if required
• Recruit Artifact User/Participants, if required
• Lesson plan or syllabus
• Instructional strategies and activities
• Participant materials
• Trainer materials
• Storyboards and scripts
• Other media
• Create model, tool, product, or program.
• Validate model, if required
• Evaluation plan (Kirkpatrick Levels 1, 2, 3, 4)
• Alpha test, Beta test, Pilots.
• Rapid Prototyping
• Participant reaction
• Trainer/facilitator reaction
• Were Design Goals met?
• Were Design Objectives met?
• Revise artifact(s), Retest, if necessary.

Stage 3: Design and Development Research aligned the NU ADE Template Process

Complete Section 3 of the ADE template presenting the study findings, conclusions, and implications. Next pull all three sections into a dissertation manuscript for approval in the ADR.

While DDR covers a wide variety of approaches, most doctoral students in the ADE program will find case study to be the preferred design. To strengthen trustworthiness of the data, multiple sources of data will typically be used.  Using multiple sources of data is called triangulation in research. Figure 1 shows examples of sources of data for DDR.

The goal is to create, use, and/or validate New Artifacts by collecting and analyzing various sources of data including:

• Existing artifacts,
• Expert individual and focus group interviews,
• Participant/user individual interviews, talk aloud-think aloud interviews, focus group interviews,
• Research observation and participant observation,
• Evaluation, Kirkpatrick Levels 1-4, and
• Needs assessment and design documents.

The new artifacts may be lesson plans, student guides, facilitator/teacher guides, learning objects, tools, models, programs, and/or products.

Figure:  Sources of Data in DDR

Ellis, T.J. & Levy, Y. (2010). A guide for novice researchers: Design and development research methods. Proceedings of Informing Science & IT Education Conference (InSITE) 2010, pp. 108-118. http://proceedings.informingscience.org/InSITE2010/InSITE10p107-118Ellis725.pdf

Goldkuhl, G. (2012). From Action Research to Practice Research. Australasian Journal of Information Systems, 17 (2). https://doi.org/10.3127/ajis.v17i2.688

Richey, R. C. & Klein, J. D. (2007). Design and Development Research. Routledge

• << Previous: Qualitative Descriptive Design
• Next: Qualitative Narrative Inquiry Research >>
• Last Updated: Jul 28, 2023 8:05 AM
• URL: https://resources.nu.edu/c.php?g=1013605

© Copyright 2024 National University. All Rights Reserved.

Privacy Policy | Consumer Information

SCIENCE & ENGINEERING INDICATORS

Research and development: u.s. trends and international comparisons.

• Report PDF (808 KB)
• Report - All Formats .ZIP (5.2 MB)
• Supplemental Materials - All Formats .ZIP (513 KB)
• MORE DOWNLOADS OPTIONS
• Share on X/Twitter
• Share on Facebook
• Share on LinkedIn
• Send as Email

Introduction

This report analyzes research and experimental development (R&D) trends in the United States and internationally. R&D refers to creative and systematic work aimed at increasing the stock of knowledge and is broken down into three categories: basic research, applied research, and experimental development (Moris and Pece 2022; OECD 2015). The Glossary section of this report summarizes key definitions.

R&D and other intangibles or intellectual property products (IPPs), such as software investment, contribute to innovation, output and productivity growth, competitiveness, and public policy goals across countries—from defense, cybersecurity, and information infrastructure to sustainable energy, environmental protection, and health (Baily, Bosworth, and Doshi 2020; CRS 2020a; NASEM 2020; Pece 2023b; OECD 2023d, 2023h). The COVID-19 pandemic impacted global science by highlighting the importance of resiliency and security in domestic and international global research networks (OECD 2022, 2023a, 2023b, 2023c, 2023d).

In the private sector, R&D is also a leading component of global value chains (GVCs) for industries at the forefront of advanced manufacturing, emerging and critical technologies, and high-technology services across the globe. International production arrangements and global R&D networks, built over the past decades (Kano, Tsang, and Yeung 2020; Papanastassiou, Pearce, and Zanfei 2020), have been challenged by pandemic-related and geopolitical factors that are impacting the organization of international R&D and economic activity and the role of critical or emerging technologies (IMF 2023; OECD 2023e).

This report is organized into four sections. The first covers U.S. R&D across the major performing and funding sectors, followed by a section on international comparisons. The last two sections focus on business R&D and federal R&D. The report also includes new information on semiconductor and other critical or emerging technologies R&D that feeds into business high-technology supply chains and public policy goals (CRS 2022a, 2022b; USG 2023).

Related Science and Engineering Indicators 2024 reports include “ Academic Research and Development ” and the forthcoming “The STEM Labor Force: Scientists, Engineers, and Skilled Technical Workers.” Three other related reports focus on production supply chains and other post-R&D activities: “ Publications Output: U.S. Trends and International Comparisons ,” “ Invention, Knowledge Transfer, and Innovation ,” and “ Production and Trade of Knowledge- and Technology-Intensive Industries .”

The principal data sources of this report are surveys and the National Patterns of R&D Resources (henceforth, National Patterns) database (NCSES 2024) from the National Center for Science and Engineering Statistics (NCSES), National Science Foundation (NSF). The Organisation for Economic Co-operation and Development (OECD) Main Science and Technology Indicators (MSTI) database (OECD 2023c) is the source for international R&D statistics. All amounts are reported in U.S. current dollars unless otherwise noted. All years are calendar years unless otherwise noted.

Related Content

This paper is in the following e-collection/theme issue:

Published on 21.5.2024 in Vol 13 (2024)

Human-Centered Design and Benefit Realization Management in Digital Health Care Solution Development: Protocol for a Systematic Review

Authors of this article:

• Kaisa Kauppinen * , MSc   ; 
• Pantea Keikhosrokiani * , PhD   ; 
• Sehrish Khan * , MSc  

Faculty of Information Technology and Electrical Engineering, University of Oulu, Oulu, Finland

*all authors contributed equally

Corresponding Author:

Kaisa Kauppinen, MSc

Faculty of Information Technology and Electrical Engineering

University of Oulu

Pentti Kaiteran katu 1

Oulu, 90570

Phone: 358 405433476

Email: [email protected]

Background: Earlier research shows that a significant number of resources are wasted on software projects delivering less than the planned benefits. It has, however, been evidenced that adopting a human-centered design approach when designing health devices can be beneficial. This understanding from earlier research has raised our interest in investigating how human-centered design might contribute to realizing the potential benefits of health care software projects. To our current knowledge, this intersection of human-centered design and benefit realization management has not yet comprehensively and consistently been researched within the context of digital health care solutions. Therefore, there is a need for evidence synthesis using systematic reviews to address this potential research gap.

Objective: The objective of this study is to examine if human-centered design helps benefit realization management processes in the development of digital health care solutions and thereby enables better benefit realization. We explore the evidence of assumed or confirmed benefits of using human-centered design in the health care domain and whether better results have been reported when the benefit realization management process is followed.

Methods: This protocol was developed following the PRISMA-P (Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols) guidelines. The proposed review would use a planned and systematic approach to identify, evaluate, and synthesize relevant and recent studies (reported in English) to see if there is evidence that using human-centered design and benefit realization management has a positive effect on realizing set benefits in those projects. We will commence a systematic literature search using human-centered design, benefit realization management, and health care–related search terms within 5 repositories (ACM Digital Library, PubMed Central, Scopus, PubMed, and Web of Science). After removing duplicate results, a preliminary scan for titles and abstracts will be done by at least 2 reviewers. Any incongruities regarding whether to include articles for full-text review will be resolved by a third reviewer based on the predefined criteria.

Results: Initial queries of 2086 records have been executed and papers are being prescreened for inclusion. The search was initiated in December 2023 and the results are expected in 2024. We anticipate finding evidence of the use of human-centered design in the development of digital health care solutions. However, we expect evidence of benefitting from both human-centered design and benefit realization management in this context to be scarce.

Conclusions: This protocol will guide the review of existing literature on the use of human-centered design and benefit realization management when developing digital health care solutions. The review will specifically focus on finding evidence of confirmed benefits derived from the use of human-centered design and benefit realization management. There may be an opportunity to gain a broader understanding of the tools or approaches that provide evidence of increased benefit realization within the health care domain.

International Registered Report Identifier (IRRID): DERR1-10.2196/56125

Introduction

Throughout the last decades, IT investments have grown significantly in all industries. Smaller portions of IT service providers´ budget s are often allocated to seeking improvements in their product or service development practices. Human-centered design (HCD) or user-centered design (UCD) belongs to this portion.

Discussion on Human-Centered Design and User-Centered Design

British Standards Institution defines HCD as “an approach to interactive systems development that aims to make systems usable and useful by focusing on the users, their needs, and requirements, and by applying human factors/ergonomics, and usability knowledge and techniques” [ 1 ]. On the other hand, R Kling [ 2 ] states UCD is an approach that bases the design, testing, and development of a product or a service on the needs of the users affected by it. UCD can thereby be described as offering a set of principles and strategies to guide the design from the perspectives of (and with input from) those humans who eventually use that product or service. When discussing UCD in the context of digital health care solutions, “user” can refer to many different groups. A user in that context can mean a clinician (eg, a doctor or a nurse), other health care professionals, care givers, or a patient. Given that, it may be more appropriate to talk about HCD, rather than UCD in the health care context, since multiple user groups are of interest here, rather than just one. In addition, Walters [ 3 ] and Steen et al [ 4 ] differentiate UCD and HCD. They argue that HCD places more emphasis on different stakeholders’ varying needs and broader contexts. This definition aligns well with the different user groups we are interested in within the health care context, particularly those who use digital health care solutions.

Digital Health Care Solution

By digital health care solutions, we mean a wide range of technologies used to improve the delivery, efficiency, and accessibility of health care services using digital tools and IT. Some examples of digital health care solutions include telehealth solutions (eg, video conferencing platforms for doctor-patient consultations), electronic health records (ie, electronic systems that store and manage patient health information), and mobile health apps (eg, medication reminder apps and mental health apps).

On Benefits and Related Disciplines

As noted by earlier research [ 5 - 7 ], a significant number of resources are wasted on software projects delivering less than the planned benefits. Furthermore, Tursunvayeva et al [ 8 ] discuss how a range of sociotechnical challenges often hamper the benefit realization processes, with many expected improvements (or benefits) either not being realized or only partially realized. These studies also discuss benefits management (BM) and explain that BM is a topic of discussion due to overlapping disciplines within BM, such as benefit realization, benefit realization management (BRM), and value management. Looking back to earlier publications on BM, BM is described as the overall process of evaluating and realizing IT benefits. Ward et al [ 9 , 10 ], on the other hand, define BRM as “the process of organizing and managing such that potential benefits arising from the use of IT are actually realized.” In addition, BRM can be described as a comprehensive management idea. The definition of BRM usually focuses on either the benefits lifecycle or the realization of potential investment benefits in change. One often used definition of BRM is “the process of organizing and managing, so that potential benefits, arising from investment in change, are actually achieved” [ 11 ]. Tursunbayeva et al [ 8 ] also discuss how benefits can be divided into 2 groups when associated with IT projects—expected benefits and realized benefits. The former can have an important role in shaping the enablement of the latter.

Health care professionals have been studied regarding their adaptation to health care technology, and the study showed that technological anxiety had a negative influence on behavioral intention [ 12 ]. Furthermore, as noted by Persson [ 13 ] in her study on technology-centered versus human-centered perspective in the design and development process, reducing the focus on technical advancements in favor of the needs of the health care user community was found to be favorable. It has also been evidenced that taking an HCD approach when designing health devices, particularly for older adults, can be beneficial and can increase the likelihood of technology acceptance [ 14 ]. Consequently, earlier research has shown that planned benefits can be misleading in digital transformation projects unless they are consistently followed up regularly throughout the process.

Establishment of Interest and Consideration

This understanding from earlier research has raised our interest in investigating how HCD might contribute to organizing, managing, and realizing the planned benefits within health care software projects. To our current knowledge, this intersection of HCD and BRM has not yet been comprehensively and consistently researched within the context of health care IT. It is, however, recognized that benefits may not appear as a direct consequence of the implementation of a digital health care solution and those benefits are necessarily not financially measurable (eg, patient or clinician experience) [ 15 ]. It is worth noting that there may be multiple reasons for the translational gap in digital health care solution development. We, therefore, value understanding and providing evidence of whether there is already an established correlation and dialogue between design techniques and software development in realizing the targeted benefits (both quantitative and qualitative) in the health care context.

Toward the goal of successfully identifying, analyzing, and summarizing these existing studies, we aim to systematically review design and development approaches in the health care domain. This includes examining their design techniques, exploration of BRM, and capabilities to realize the planned benefits. The theoretical assumption of this study is that when HCD is used in digital health care solutions’ development projects, the set and planned benefits are better realized. Additionally, it will be controversially considered whether we can assume that a technology-centered approach in the health care sector has contributed to the realization of disbenefits.

Research Questions

In this paper, we examine if HCD helps the BRM process in the development of digital health care solutions , thereby enabling better benefit realization.

To address our aims, we have set the following research questions:

• Is HCD used in the design and development of software solutions for health care domain? What are the assumed or confirmed benefits of HCD?
• Is BRM used in the process of developing health care software solutions? Have better results been reported when BRM is included in the process?
• Have there been studies on whether HCD helps the BRM process in the development of digital health care solutions? What is the evidence that planned benefits have been better realized with the help of HCD?

To answer our research questions, we have identified the following objectives for the study:

• to explore the evidence of assumed or confirmed benefits of using HCD in the development of digital health care solutions.
• to investigate whether there is evidence that better results have been reported when BRM is included in the development process of digital health care solutions.
• to identify whether there is evidence that planned benefits have been better realized with the help of HCD and BRM in health care context.

Design of the Study

For the design of the systematic review, we will use the PRISMA-P (Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols) guidelines [ 16 ]. PRISMA-P consists of a literature search, article selection and screening, data extraction and analysis, as well as an assessment of study quality and bias. Figure 1 depicts the study structure, which is explained in more detail in the next sections.

Literature Search

We will survey 5 large databases of digital literature—ACM Digital Library, PubMed Central, Scopus, PubMed, and Web of Science—using keywords and terms presented in Table 1 .

Search strings need to be adapted to suit the specific requirements of the different databases and electronic libraries.

Search terms focus on 3 key areas: HCD terms (eg, human-centered and user-centered), BRM terms (eg, benefit management, benefit realization management, and benefit realization), and health care–related terms (eg, electronic health record and health care IT). An initial digital library search has provided 2086 papers.

Eligibility Criteria

Eligibility will be described by defining the criteria for including articles in the review and the criteria for excluding found publications from further processing. Studies will be selected according to the criteria described in Textbox 1 .

Inclusion criteria

• Only peer-reviewed articles, conference proceedings, and book chapters.
• Papers that describe only quantitative, qualitative, or mixed methods.
• Papers that describe the use of human-centered design methodologies when designing digital health care solutions, such as co-design, cocreation, design sprint, or usability testing.
• Papers that describe the use of benefit realization management when developing digital health care solutions.
• Papers that describe if and how benefits have been realized with the help of human-centered design or benefit realization management when developing digital health care solutions.

Exclusion criteria

• Papers that are purely study protocols, systematic literature reviews, abstracts, posters, short papers, or scoping reviews.
• Papers that are not in English.
• Papers that are not research articles or publication or are otherwise off topic.
• Papers not related to digital health care solutions.
• If human-centered design or benefit realization management is explained but not applied in the digital health care solution.
• Papers that discuss fitness applications, well-being applications, and sport applications (we will only focus on applications and solutions related to therapeutic areas, such as cardiology, neurology, or oncology, that directly affect patient care, either through use by clinicians or together with patients’ care givers; therefore, in this review, we will exclude more consumer-targeted applications).

Article Selection and Screening

After implementing the search strategy on the named databases, the results will be imported into Covidence (Veritas Health Innovation) to initiate the selection and screening process. Covidence automatically identifies duplicates and supports reviewers to execute their screening simultaneously and independently.

In the initial search and screening, the focus will be on titles and abstracts only. Three people will be involved in the screening. Each paper will be reviewed by at least 2 independent reviewers. All reviewers have backgrounds in digital health and information processing science. After completing the initial search and screening phase of the study, reviewers will move on to screening the full texts of the selected articles.

If the reviewers face disagreement in either the initial screening or during the full-text screening, reasons for disagreement will be discussed. If no resolution is found during the discussions, a fourth reviewer may act as an arbitrator to review the disagreed article and decide whether the article will be included or excluded. The disagreement will be documented together with the eventual outcome and its reason.

Data Extraction and Analysis

The key information of the included articles will be extracted using the Data Extraction Form function available on Covidence. The extraction task will be completed by the 2 initial reviewers. The data extraction form within Covidence will assist in the extraction of relevant information from the selected studies.

Assessment of Study Quality and Bias

Quality assessment will be produced in parallel to the data extraction process using a quality assessment form that will be included in Covidence. The quality assessment form will follow the checklist proposed by Dybå and Dingsøyr [ 17 , 18 ]. The checklist highlights the following main focus areas:

• The paper is based on research and research aims are clearly stated.
• Context, where research was carried out, is adequately described.
• The research design and recruitment strategy were appropriate for the aims of the research.
• The data were collected to address the research issues, and the data analysis was rigorous enough.
• The study is of value for research or practice.

Data Synthesis

The included studies may be a collection of qualitative and quantitative data that will need to be transformed into a qualitative format. For data synthesis, narrative synthesis will be used, and therefore, a narrative form with an appropriate table format will be used. The table will consist of categories, such as the use of HCD, the use of BRM, evidence of assumed benefits (quantitative or qualitative), therapeutic areas (eg, cardiology, neurology, or oncology), modality of digital user interfaces (eg, desktop, mobile, or smartwatch), targeted user groups (eg, clinicians, patients, or care givers).

With the help of thematic coding, we will organize the extracted data into categories based on similarities and patterns. The categories will be further developed into a framework that will be used to construct a narrative addressing the proposed research questions and objectives of the review. Overall, the entire search strategy from the search and article selection to data synthesis will be piloted with 3-5 studies to address any need to change the approach.

As of January 2024, we have identified 2086 papers that have met our initial screening criteria. These papers are now being further analyzed to exclude those that do not exactly match our inclusion criteria described in Textbox 1 .

The objective of this study specifically is to explore whether there is evidence that the use of HCD in the development of digital health care solutions helps the BRM process and enables better benefit realization.

Based on the initial search results and incomplete full paper screening, we can already see that there is evidence of using HCD in the creation of digital health care solutions, and by doing so, the benefits of the chosen design principle have been reported. However, the actual use of BRM in the context of digital health care solutions’ development does not seem so evident.

The large number of found papers in the initial search may be considered as a limitation of the study, potentially resulting in a broad sequence of alternative perspectives. We plan to critically review the inclusion and exclusion criteria while screening the remaining abstracts and titles and when reviewing selected full papers. This approach will increase the study’s quality level.

This review will enable the critical appraisal and synthesis of evidence on the successful use of HCD and BRM when developing digital health care solutions, leading to a better realization of the benefits set at the beginning of those projects. Controversially, if there is no evidence of both activities and causation, there is an opportunity for new knowledge creation going forward.

Acknowledgments

We would like to acknowledge the Tauno Tönning fund for financing the development of this work.

Authors' Contributions

KK designed the study and wrote the original draft; PK and SK contributed to the content and approved the final manuscript.

Conflicts of Interest

None declared.

• Ergonomics of human-system interaction - Human-centred design for interactive systems. Bsi.knowledge. URL: https:/​/knowledge.​bsigroup.com/​products/​ergonomics-of-human-system-interaction-human-centred-design-for-interactive-systems/​tracked-changes [accessed 2024-12-10]
• Kling R. The organizational context of user-centered software designs. MIS Quarterly. Dec 1977;1(4):41. [ CrossRef ]
• Walters PJ. Knowledge in the making: prototyping and human-centred design practice. URL: https://shura.shu.ac.uk/20492/1/10701139.pdf [accessed 2024-01-10]
• Steen M, Koning ND, Pikaart A. Exploring human centred approaches in market research and product development: three case studies. 2004. Presented at: Dutch HCI '04: Proceedings of the Conference on Dutch Directions in HCI; June 10; Amsterdam, The Netherlands. [ CrossRef ]
• Holgeid KK, Jørgensen M, Sjøberg DIK, Krogstie J. Benefits management in software development: a systematic review of empirical studies. IET Software. Feb 03, 2021;15(1):1-24. [ CrossRef ]
• Svejvig P, Schlichter BR. The long road to benefits management: toward an integrative management model. Proj Manag J. Feb 11, 2020;51(3):312-327. [ CrossRef ]
• Serugga J, Kagioglou M, Tzortzopoulos P. Front end projects benefits realisation from a requirements management perspective—a systematic literature review. Buildings. Apr 28, 2020;10(5):83. [ CrossRef ]
• Tursunbayeva A, Bunduchi R, Pagliari C. “Planned benefits” can be misleading in digital transformation projects: insights from a case study of human resource information systems implementation in healthcare. SAGE Open. Jun 24, 2020;10(2):1-10. [ CrossRef ]
• Ward J, Taylor P, Bond P. Evaluation and realisation of IS/IT benefits: an empirical study of current practice. Eur J Inf Syst. Feb 01, 1996;4(4):214-225. [ CrossRef ]
• Ward J. In: Ward J, Daniel E, editors. Benefits Management: How to Increase the Business Value of Your IT Projects, Second Edition. John Wiley & Sons. New York, NY. Wiley; 2012.
• Bradley G. Benefit Realisation Management: A Practical Guide to Achieving Benefits Through Change. Second Edition. New York. Routledge; 2016.
• Walters LEM, Scott RE, Mars M. Exploration of benefits realisation management for teledermatology scale-up framework development and sustainable scaling. IMU. 2019;15:1-6. [ CrossRef ]
• Persson J. A review of the design and development processes of simulation for training in healthcare - a technology-centered versus a human-centered perspective. Appl Ergon. Jan 2017;58:314-326. [ CrossRef ] [ Medline ]
• Harte RP. A Human-Centered Design of a Connected Health System for Older Adults. Galway. National University of Ireland, Galway; Aug 15, 2017.
• Cresswell K, Coleman J, Smith P, Swainson C, Slee A, Sheikh A. Qualitative analysis of multi-disciplinary round-table discussions on the acceleration of benefits and data analytics through hospital electronic prescribing (ePrescribing) systems. J Innov Health Inform. Jul 04, 2016;23(2):501-509. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. PRISMA-P Group. Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. Jan 02, 2015;350:1-25. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yang L, Zhang H, Shen H, Huang X, Zhou X, Rong G, et al. Quality assessment in systematic literature reviews: a software engineering perspective. Inf Softw Technol. Feb 2021;130:1-24. [ CrossRef ]
• Dybå T, Dingsøyr T. Empirical studies of agile software development: a systematic review. Inf Softw Technol. Aug 2008;50(9-10):833-859. [ CrossRef ]

Abbreviations

Edited by S Ma; submitted 20.01.24; peer-reviewed by S Budhdeo, A Hassan; comments to author 04.04.24; revised version received 16.04.24; accepted 16.04.24; published 21.05.24.

©Kaisa Kauppinen, Pantea Keikhosrokiani, Sehrish Khan. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 21.05.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.

More From Forbes

5 highest paying skills for project managers in 2024, from research.

• Share to Facebook
• Share to Twitter
• Share to Linkedin

To remain competitive and gain an edge as a project management professional, it helps to learn one ... [+] or more of these five critical skills this year

Within a competitive job market in which 25 million project managers are projected to be in demand over the next few years, it's critical for aspiring and existing project management professionals to stay ahead of the curve so they can secure the highest-paying roles. While the project management profession in itself is a well-remunerated industry, with U.S. median salaries jumping to as high as $120,000, that doesn't necessarily mean that you can sit back, relax, and assume you're worthy of a six-figure salary.

There are things that lie within your power, which you can do to dramatically skyrocket your earnings.

When exploring how to increase and even multiply your salary as a project manager, it's best to look into what are the highest-paying, in-demand skills within the project management industry, so that you can position yourself as best suited for a promotion, or as a star candidate when job-searching.

Highest-Paying In-Demand Skills For Project Managers

The compensation and salary data company, PayScale, compiled a list of high-paying skills which include a combination of hard, technical skills, and soft or power skills for those within the project industry.

It's worth noting that there are some industries in which the impact of these skills on your pay increase will be more visible than others. For example, considering programme/project management within the technology industry, you can expect to earn more for your skill set, as the tech industry is renowned for offering the highest salaries to project and program management professionals. As such, some of the skills listed below are more specific to the technology industry.

Best High-Yield Savings Accounts Of 2024

Best 5% interest savings accounts of 2024.

However, many of these skills are equally in demand in other industries as well, so they can still be applicable and relevant to you.

These competencies include:

1. Agile Software Development

Using the agile software development approach for building and delivering software products entails prioritising flexibility responsiveness and adaptability to change as well as collaboration. Following this framework, you and your team would be working in sprints, (short iterations) which allows you to make incremental progress and quickly respond to stakeholder requirements and feedback to meet their needs.

This is the highest-paying skill for project managers, with Payscale data revealing that it can boost your earnings by 47%. Coursera and Skillsoft are two of the amazing resources that are available for you to learn this skill.

The tech industry offers the highest salaries for PMs

2. Business Analysis

Analytical thinking is listed in the World Economic Forum's Future of Jobs Report 2023 as the number one skill needed by professionals over the next few years.

Business analysis, in simple terms, is when you identify business needs and research solutions for them. This skill comes handy in your PM role, as you will naturally be working with stakeholders on a regular basis to gather and analyze data on their needs and define requirements, and then propose the best solution with your program or project. This skill improves salary prospects by up to 43%.

3. Engineering Design

Next on the list is engineering design, which is of course more specifically focused on the technology industry. This involves taking the results of your business analysis, and creating detailed plans and specifications, including prototyping and testing, to ensure you deliver a high-quality product that satisfies your stakeholders' needs. This skill is more relatable to product managers, and provides a 14% uptake in salary.

4. Risk Management/Risk Control

With all the changes that are occurring to disrupt industries in 2024, it's no wonder that risk management and risk control are highly in-demand skills for project and program managers to possess. This skill also has the potential for a 14% salary boost, according to Payscale analysis.

You can learn more about how to comprehensively manage and plan for risk by undertaking courses and certifications such as those offered by the Institute of Risk Management, which are globally recognized.

5. Strategy

Strategic thinking is a trademark of leadership, and of exceptional program and project management. You need to be able to steer your project team in the right direction to achieve successful project outcomes, and this means you'll need to have a solid strategy that is clearly articulated so that everyone is on the same page. Setting clear goals and objectives, and assigning the right team members to fulfil each task according to their strengths requires some practice to get it right, but it can be achieved if you're persistent and intentional. Possessing and showcasing this skill results in a 12% salary increase.

Risk management skills helps ensure your project is prepared for worst-case scenarios and minimizes ... [+] negative outcomes, including those that could affect your job directly

By investing in yourself this year by upskilling in one or more of these five essential competencies, you can position yourself to successfully meet the evolving demands of your stakeholders and the project management industry, and unlock numerous career-building and salary-boosting opportunities.

• Editorial Standards
• Reprints & Permissions

Join The Conversation

One Community. Many Voices. Create a free account to share your thoughts. 

Forbes Community Guidelines

Our community is about connecting people through open and thoughtful conversations. We want our readers to share their views and exchange ideas and facts in a safe space.

In order to do so, please follow the posting rules in our site's  Terms of Service.   We've summarized some of those key rules below. Simply put, keep it civil.

Your post will be rejected if we notice that it seems to contain:

• False or intentionally out-of-context or misleading information
• Insults, profanity, incoherent, obscene or inflammatory language or threats of any kind
• Attacks on the identity of other commenters or the article's author
• Content that otherwise violates our site's  terms.

User accounts will be blocked if we notice or believe that users are engaged in:

• Continuous attempts to re-post comments that have been previously moderated/rejected
• Racist, sexist, homophobic or other discriminatory comments
• Attempts or tactics that put the site security at risk
• Actions that otherwise violate our site's  terms.

So, how can you be a power user?

• Stay on topic and share your insights
• Feel free to be clear and thoughtful to get your point across
• ‘Like’ or ‘Dislike’ to show your point of view.
• Protect your community.
• Use the report tool to alert us when someone breaks the rules.

Thanks for reading our community guidelines. Please read the full list of posting rules found in our site's  Terms of Service.

Implementing Research-Based Learning Design in Edmentum Courseware

At Edmentum, we regularly review learning sciences and education research to design with the latest findings in mind. This helps us ensure that our products deliver high quality learning acceleration for all students as knowledge about “what works” in the field evolves.

Our Learning Design team recently conducted a review of research to support an upcoming update of our Courseware Learning Design and Research Base . In this article, we share a preview into how research continues to inform our learning design principles while highlighting how we’ve been applying this research to recent Courseware development.

Edmentum Courseware Learning Design Principles

Edmentum Courseware’s learning methodology is rooted in six learning design principles that provide the bedrock for our development. Our review of recent research confirms that each principle maintains a sound basis in theory, best practice, and research.

Mastery Learning: Focus on efficient learning. Students show mastery of key content, invest time on not-yet mastered content, and proceed through learning new content at their own pace.

Active, Engaging Learning: Involve learners in responding to and manipulating information while they learn — ensuring their involvement in building understanding and minimizing passive reception of information.

Deliberate Practice: Offer intentional, structured, and sustained practice that builds thoughtfully in complexity to support increasing levels of understanding.

Explicit Instruction: Support successful learning by providing clear skill statements, modeling of learning outcomes, and reducing cognitive load.

Scaffolding: Buoy learning by providing specific supports when learners need them and systematically removing them—leading to independence as learners approach mastery.

Metacognitive Strategies: Engage students in reflecting on how they best learn and evaluating their thought processes to help themselves along their learning path.

Recent Research Informing Edmentum Courseware

Edmentum ensures the quality of our product design in part by learning from rigorous research shared in a variety of credible venues, including peer-reviewed academic journal articles and books from academic publishers. Below are five such sources included in our recent review, and for each, we offer examples of what the research looks like in the design of courses across the Courseware catalog .

1. Effect Sizes in Student Achievement Research

John Hattie updated his influential meta-analysis of evidence-based education research about student achievement, resulting in his 2023 publication of Visible Learning: The Sequel; A synthesis of over 2,100 meta-analyses relating to achievement . This book offers numerous implications for our design, given the breadth of the research he reviewed. Key insights we noted in this update to Hattie's work include the importance of:

• reducing the cognitive load for learners,
• providing detailed direct instruction, and
• including worked examples to enhance learners' understanding of concepts.

How Edmentum Applies this Research to the Design of Courseware

Edmentum course designers intentionally reduce the cognitive load for learners in various ways through explicit, direct instruction in all courses. Every Courseware lesson opens with one or more learning objective, ensuring learners and educators know the focus of attention for the lesson.

Figure 1. The new Web Technologies course for secondary students starts each tutorial with a clearly stated objective detailing what students can expect to learn.

New content is introduced in well-defined chunks and all courses are designed to include direct instruction when teaching new concepts and/or procedures. This includes explicit call out and defining of relevant academic vocabulary during course instruction. Here’s what this looks like in the Web Technologies course in our updated career and technical education (CTE) catalog .

Figure 2. Also in the Web Technologies course, students have relevant vocabulary called out and defined as new concepts get explained in each tutorial.

Edmentum Courseware supports learning by showing worked examples of key concepts. Students can self-pace through complex examples to reduce cognitive load and increase understanding. For example, all Mathematics courses include worked examples allowing learners to see the thinking processes involved in solving problems.

Figures 3a-d. In a recent update to the integrated Math 2 course for North Carolina, high school learners can go at their own pace through a worked example of how a nonlinear equation can be solved algebraically.

Our English courses also include worked examples for concepts being taught in the lesson. Here we see modeling of literary analysis and analyzing structures in informational texts.

Figure 4. In an English 9 tutorial, learners see a worked example of understanding a character’s perspective in literature.

Figure 5. In another English 9 tutorial, learners see a worked example analyzing the structural elements of an informational text and using highlighting to reduce the cognitive load for learners who can attend to the color-coding.

2. Comparing Active Learning with Passive Instruction

In another meta-analysis of 144 experimental or quasi-experimental studies capturing insights from more than 10,000 elementary, middle, and high school participants, researchers explored comparisons of an active learning treatment with groups who received traditional lecture-based instruction. Researchers found "that active learning is more effective on students' attitudes toward the course" and has the potential to "increase academic achievement" ( Tutal & Yazar, 2022, p. 14 ), demonstrating the importance of including student-centered activities that engage learners with content in many different ways.

Figure 6. Direct instruction in a lesson in the 7th grade Science course, giving students relatable examples of systems.

Edmentum embeds a variety of active learning strategies throughout all Courseware courses to involve learners in responding to and manipulating information while they learn. Students receive regular zero-stakes opportunities to confirm their understanding during instruction. We see this in action in the self-checked activities and formative assessments students receive in a 7th grade Science course, for example. In this lesson, students learn how energy is transformed within a system, through direct instruction and a series of ungraded checks for understanding.

Figure 7. This is one of the zero-stakes checks for understanding that ask students to respond to and apply what they’re learning about to the roller coaster system in this self-checked activity.

In our Mathematics courses, students also get practice applying concepts to real-world situations as they learn. Here again, we see the roller coaster used as an example in the lesson, this time to help Algebra 2 students understand applications for the piecewise functions they are learning about.

Figure 8. In a version of Algebra 2 redesigned to align to the new Florida B.E.S.T. standards, students see how piecewise functions can be used to model real-world situations, like the path of a roller coaster.

Figures 9a-d. Then, the next part of this lesson allows students to go at their own pace to analyze the graph of this function separately.

Students are then able to practice on their own, again in an ungraded activity that asks them to apply their learning about piecewise functions to a different real-world scenario and they receive optional additional supports in the form of a worked example with yet another application of these functions. All of which helps students engage more actively and see multiple applications of what they are learning.

Figure 10a.

Figure 10b.

Figures 10a-b. In this self-checked formative assessment, students in the Algebra 2 course now apply what they are learning about piecewise functions to the real-world example of population change (a). They can choose to explore the worked example (b) to see how to sketch a piecewise function to explain the path of an elevator.

3. Effective and Ineffective Digital Scaffolding in Credit Recovery Courses

Many of Edmentum’s partners use Courseware to provide students who previously failed a course with a learning option to earn necessary credits. As such, we sought to understand how best to support learning in these unique contexts from recent research, including a mixed methods study of high school students enrolled in a credit recovery course on environmental science ( Kim, et al., 2022 ). This research offered positive and negative results related to various types of scaffolding, including conceptual, metacognitive, and strategic scaffolding. The researchers' presentation of how scaffolding not only supported some learners with prompts for next steps, for example, but also frustrated other learners with more information than they needed or wanted, offered direction for the kinds of prompts to include while not overdoing such prompting.

Edmentum courses include guided notes and audio and visual supports that provide scaffolding to ensure that learners can engage with and retain information from the content of lessons.

Figure 11. This page from the Guided Notes in one lesson in the new Ethnic Studies course showcases the optional scaffolding available to students to help them organize and keep track of what they learn in Edmentum courses.

Figure 12. Students receive metacognitive support through this video in one of the early lessons in the Ethnic Studies course. In recognition of the social and emotional learning needs when discussing difficult content, this video provides a scaffold for students to manage their feelings to mitigate disruptions to their learning in the course.

Courses also include prompts embedded within the instruction that are designed to help students strengthen their conceptual understanding, engage in metacognitive reflection, and strategically organize their learning. In many cases, these scaffolds are optional, so students can use none, some, or all as their learning needs require. Here are some examples of such scaffolding from two English courses.

Figure 13. Learners in the English 10 course receive metacognitive scaffolding supports midway through a tutorial as they practice analyzing literature. As optional clicks for the additional information, students can opt to use these scaffolds or not.

Figure 14. In English 9, the introduction of Unit 4 recaps highlights of the previous units and offers strategic scaffolding to guide learners to think about the informational texts they read and evaluated while preparing their culminating research paper.

4. Lessons Learned about Online Instructional Delivery for K-12

As the COVID-19 pandemic began, many educators and researchers recognized a general lack of understanding about what practices best support K-12 learning in online settings. To meet that need, Johnson and colleagues (2023) conducted a systematic review of 251 research articles describing online delivery of instruction in K-12 schools in the United States. While this review has multiple implications for the design of our courses, highlights from the findings include:

• “More frequent self-reflection is associated with improved learning outcomes” (p. 376).
• “Evidence suggests that feedback in online instruction supports student learning when it is frequent and meaningful…[and when it] includes elaborated explanations as opposed to simply indicating, for example, that a response is correct or incorrect” (p. 382).
• “Strategically facilitating discourse between teachers and students and between peers in online settings is an important way to support students’ meaning-making processes” (p. 380).

Edmentum Courseware embeds structured self-reflection opportunities throughout our courses. Some of our most recent courses, including Ethnic Studies (which will be available for the start of the 2024-2025 school year), demonstrate various elements of Courseware that allow students to frequently engage in self-reflection.

Figure 15a.

Figure 15b.

Figure 15c.

Figures 15a-c. In the first unit of the Ethnic Studies course, students learn about foundational concepts, including bias (a). That lesson ends by engaging in a self-checked opportunity for reflection about their understanding these concepts (b) as they make connections to what they learned about bias (c).

Figure 16. In the Guided Notes accompanying the Ethnic Studies course, students are asked to respond to what they have learned as they make personal connections to the material along their learning journey.

Edmentum Courseware also provides students with immediate feedback on formative checks for understanding. As this example from a Chemistry course shows, that feedback offers explanations for why an answer is correct to make the feedback more meaningful to learners.

Figure 17. When students select the correct answer in this formative assessment in a Chemistry tutorial, they receive a meaningful explanation for why that response answers the question. This same explanation also appears after two incorrect attempts at answering the question.

Lastly, Edmentum Courseware often includes discussions between students to help facilitate meaning-making. We see this in recent updates to our Mathematics courses.

Figure 18. After students in the Florida Algebra 1 course collect and model their own set of bivariate data about a topic of their choice, this activity asks them to present their model to peers and lead a discussion related to the data.

Figure 19. In a middle school Mathematics 6 course, students use an online discussion board to respond to prompts that ask them to apply what they learned and then choose a classmate’s problem to solve and explain their thinking.

5. Supporting Autonomy Positively Impacts Online Engagement

The need for emergency remote instruction brought about by the pandemic generated a great deal of research about online learning in recent years. One such study interviewed students and teachers to explore student engagement and found that online learning environments that support autonomy help engage students cognitively in self-regulated learning ( Chiu, 2023 ).

Edmentum courses provide students autonomy and choice as they learn, helping them make personal connections to facilitate learning and increase engagement. We see this when students are given the autonomy to choose research topics of personal interest, as in this middle school Life Science animal research activity.

Figure 20a.

Figure 20b.

Figures 20a-b. In a Life Science course, middle school students learn about animals and then have the autonomy to apply what they learned with research about an animal of their choice (a). They then have a creative opportunity to display their research in an infographic, as this octopus sample answer in the course depicts (b).

Similarly, in the Career Explorations course that was recently released, students exercise autonomy as they determine career clusters of interest and learn more about how to enter specific careers of their choice.

Figure 21. The Career Explorations course ends Unit 1 with an activity that gives students the autonomy to explore careers of interest and to learn about the requirements are for entering each career pathway.

Figure 22. Students in this same Career Explorations course then research where in their state they might be able to pursue the education needed to enter their chosen pathways.

Looking Ahead

This article has shared examples that illuminate how Edmentum applies the latest academic research to set high expectations for students, to engage them in high quality learning acceleration, and to ensure that good learning design is embedded throughout our Courseware catalog to support their success. We will continue our ongoing review of research, updating our courses to maintain these commitments to innovation and excellence.

Visit our Courseware product page for additional information or get a quote .

Get the latest education insights sent directly to your inbox

Subscribe to our knowledge articles.

This paper is in the following e-collection/theme issue:

Published on 21.5.2024 in Vol 8 (2024)

This is a member publication of Open University

A Web-Based Intervention to Support the Mental Well-Being of Sexual and Gender Minority Young People: Mixed Methods Co-Design of Oneself

Authors of this article:

Original Paper

• Katherine Brown 1 , PhD   ; 
• Mathijs F G Lucassen 2 , PhD   ; 
• Alicia Núñez-García 3 , PhD   ; 
• Katharine A Rimes 4 , DPhil, DClinPsy   ; 
• Louise M Wallace 3 , PhD   ; 
• Rajvinder Samra 3 , PhD  

1 Centre for Research in Psychology and Sports Science, School of Life and Medical Sciences, University of Hertfordshire, Hatfield, United Kingdom

2 School of Health & Psychological Sciences, City, University of London, London, United Kingdom

3 School of Health, Wellbeing & Social Care, The Open University, Milton Keynes, United Kingdom

4 Institute of Psychiatry, Psychology & Neuroscience, Kings College London, London, United Kingdom

Corresponding Author:

Katherine Brown, PhD

Centre for Research in Psychology and Sports Science

School of Life and Medical Sciences

University of Hertfordshire

College Lane

Hatfield, AL109AB

United Kingdom

Phone: 44 1707 284 615

Email: [email protected]

Background: Sexual and gender minority youth are at greater risk of compromised mental health than their heterosexual and cisgender peers. This is considered to be due to an increased burden of stigma, discrimination, or bullying resulting in a heightened experience of daily stress. Given the increasing digital accessibility and a strong preference for web-based support among sexual and gender minority youth, digital interventions are a key means to provide support to maintain their well-being.

Objective: This paper aims to explicate the co-design processes and underpinning logic of Oneself , a bespoke web-based intervention for sexual and gender minority youth.

Methods: This study followed a 6-stage process set out by Hagen et al (identify, define, position, concept, create, and use), incorporating a systematic scoping review of existing evidence, focus groups with 4 stakeholder groups (ie, sexual and gender minority youth, professionals who directly support them, parents, and UK public health service commissioners), a series of co-design workshops and web-based consultations with sexual and gender minority youth, the appointment of a digital development company, and young adult sexual and gender minority contributors to create content grounded in authentic experiences.

Results: Oneself features a welcome and home page, including a free accessible to all animation explaining the importance of using appropriate pronouns and the opportunity to create a user account and log-in to access further free content. Creating an account provides an opportunity (for the user and the research team) to record engagement, assess users’ well-being, and track progress through the available content. There are three sections of content in Oneself focused on the priority topics identified through co-design: (1) coming out and doing so safely; (2) managing school, including homophobic, biphobic, or transphobic bullying or similar; and (3) dealing with parents and families, especially unsupportive family members, including parents or caregivers. Oneself’s content focuses on identifying these as topic areas and providing potential resources to assist sexual and gender minority youth in coping with these areas. For instance, Oneself drew on therapeutic concepts such as cognitive reframing, stress reduction, and problem-solving techniques. There is also a section containing relaxation exercises, a section with links to other recommended support and resources, and a downloads section with more detailed techniques and strategies for improving well-being.

Conclusions: This study contributes to research by opening up the black box of intervention development. It shows how Oneself is underpinned by a logic that can support future development and evaluation and includes diverse co-designers. More interactive techniques to support well-being would be beneficial for further development. Additional content specific to a wider range of intersecting identities (such as care-experienced Asian sexual and gender minority youth from a minority faith background) would also be beneficial in future Oneself developments.

International Registered Report Identifier (IRRID): RR2-10.2196/31036

Introduction

Worldwide, it is estimated that up to 10% of the adolescent population identifies as being either a sexual or gender minority youth; that is, they identify as lesbian, gay, bisexual, transgender, queer, or another sexual or gender minority (lesbian, gay, bisexual, transgender, and queer [LGBTQ+]) [ 1 - 3 ]. Sexual and gender minority youth are known to be at greater risk of poor mental health than their heterosexual and cisgender peers [ 1 , 4 ]. This elevated risk is suggested to be largely related to an increased burden of stigma, discrimination, or bullying resulting in a heightened experience of stress in their day-to-day lives [ 5 , 6 ]. Clearly, work needs to continue to improve social environments for sexual and gender minority youth to reduce the additional stress they experience, but this will take time. In parallel, research is needed to identify what can be done to support sexual and gender minority youth to protect their mental health and well-being and help them build the skills and resilience they will need to thrive. This is increasingly important for the youngest sexual and gender minority youth as there is evidence suggesting that they are coming out at an earlier age than previous generations [ 6 , 7 ]. Their younger age may mean that they have had less time and opportunity to develop strong support networks and coping skills compared with those who come out at an older age [ 6 , 8 ].

Current and recent generations of young people have grown up in the digital age. Often referred to as digital natives [ 9 ], they have only experienced a world with access to the internet [ 9 ]. The latest data suggest that almost all homes in the United Kingdom have access to the internet [ 10 ] and 97% of individuals aged 12 to 15 years have their own mobile phone, with the vast majority using it to access the internet [ 11 ]. Young people are also known to spend much of their time in web-based spaces, which can assist their early attempts to seek information or obtain support on the issues they face. Similarly, a UK Department of Health and Social Care–commissioned report highlighted a strong preference among sexual and gender minority youth to access help on the internet, whereby 82.3% (n=572) of sexual and gender minority youth participants reported being “likely” or “very likely” to choose support in this format [ 12 ]. For this reason, providing web-based resources to support sexual and gender minority youth and the adults who assist them could be a widely accessible and relatively low-cost public health approach to improving their health and well-being.

In this paper, we present the detailed systematic steps we took to develop Oneself, a bespoke digital web-based resource to support sexual and gender minority youth regarding some of the most pressing challenges associated with growing up and being a sexual and gender minority young person. Drawing on the “identify, define, position, concept, create, and use” stages set out by Hagen et al [ 13 ] for participatory design with young people in mental health promotion, the process initially involved a scoping review of the strategies used in existing interventions [ 14 ]; in-depth interviews with adult experts who support sexual and gender minority youth, including parents; and focus groups with sexual and gender minority youth. We then engaged in a co-design process involving workshops with sexual and gender minority youth to determine priorities for the focus of the content and the look and feel of the resource and develop aspects of the content itself.

In addition to drawing on evidence from the scoping review, we drew on the firsthand expertise of sexual and gender minority youth as participatory research and co-design with intended end users of interventions are essential for their optimization in pragmatic terms. For example, knowledge about the needs of unique subpopulations may be limited, and co-design processes can help enhance an intervention’s acceptability [ 15 - 18 ]. In instances in which a group is frequently marginalized, such as sexual and gender minority youth, co-design is especially important because it represents a way to empower and democratize research and its outputs [ 19 ]. Co-design with underserved populations, including sexual and gender minority youth, allows pertinent diversity considerations to be addressed, for instance, factors regarding language, symbols, and character use in digital mental health technologies [ 20 ]. Hence, co-design processes are an attempt to help inform the creation of acceptable resources and assist in not only avoiding further alienating populations such as sexual and gender minority youth but also offering them a voice and greater inclusion. The approach applied by Hagen et al [ 13 ] was specifically chosen because it has been applied successfully in the past to support sexual and gender minority youth in terms of their mental health. Making intervention development processes replicable and transparent in how they are intended to bring about change for end users is also recognized as important for developing the science of health and well-being [ 21 ]. With this in mind, we outlined what we planned to do at the start of our project in our published study protocol [ 6 ]. This protocol was submitted in June 2021, before the project officially commenced.

This paper sets out the systematic stages involved in developing Oneself for sexual and gender minority youth and describes how the findings or outcomes from each stage fed into content development and refinements. It also aims to clearly explicate how each feature and its content are intended to support sexual and gender minority youth and promote change so that any future research involving Oneself can incorporate evaluation against the logic that underpins it.

In accordance with our published protocol [ 6 ], we set out to follow the stages in intervention co-design as outlined by Hagen et al [ 13 ]. Intervention development and co-design are rarely a straightforward, linear process. In practice, some tasks need to happen in parallel, and researchers and coproducers may need to cycle back and repeat elements of the process as additional challenges emerge and new insights arise. The 6 stages of co-design involving adult experts and sexual and gender minority youth are set out below with a brief description of project activities involved in developing the resource aligned to that stage and links to the relevant methods section where more detail is provided ( Textbox 1 [ 13 ]).

• Focus groups with sexual and gender minority youth (see the Interviews and Focus Groups With Sexual and Gender Minority Youth, Adult Experts, and Parents: Identify and Define Stages section)
• Systematic scoping review (see the Systematic Scoping Review: Identify and Define Stages section)
• Interviews with adult experts and parents (see the Interviews and Focus Groups With Sexual and Gender Minority Youth, Adult Experts, and Parents: Identify and Define Stages section)
• Team co-development to finalize decisions and solutions (see the Findings of the Research Team’s Co-Design Meetings in June 2022: Design, Position, and Concept Stages section)
• Initial co-design workshops with sexual and gender minority youth and email and web-based consultation (see the Initial Co-Design Workshops With Sexual and Gender Minority Youth and Email or Web-Based Consultations: Position and Concept Stages section)
• Appointment of digital developer (see the Appointment of Digital Developer [Preparation for Delivering the Concept, Create, and Use Stages] section)
• Team co-development to finalize decisions and solutions (see the Research Team Co-Development to Finalize Decisions on the Focus and Topic Areas [Define, Position, and Concept Stages] section)
• Initial co-design workshops with sexual and gender minority youth and email or web-based consultation (see the Initial Co-Design Workshops With Sexual and Gender Minority Youth and Email or Web-Based Consultations: Position and Concept Stages section)
• Questionnaire to assess look and feel design options (see the Initial Co-Design Workshops With Sexual and Gender Minority Youth and Email or Web-Based Consultations: Position and Concept Stages section)
• Appointment of sexual and gender minority community members through specialist media and modeling agencies (see the Appointment of Sexual and Gender Minority Contributors Through Specialist Media and Modeling Agencies: Concept Stage section)
• Further co-design workshops with sexual and gender minority youth (see the Further Co-Design Workshops With Sexual and Gender Minority Youth: Create Stage section)
• Filming with sexual and gender minority contributors (see the Introducing the Sexual and Gender Minority Contributors: Create Stage section)
• Development work by appointed digital provider (see The Oneself Resource section)
• Feedback from think aloud user interviews (MFG Lucassen, unpublished data, 2024)
• Feedback from adult expert interviews (MFG Lucassen, unpublished data, 2024)

Ethical Considerations

Ethics approval for the aspects of the study involving human participation was granted by the Human Research Ethics Committee at The Open University (OU) before data collection began (ethics approval HREC/4059/Lucassen). All participants, both adults (eg, professionals who directly support sexual and gender minority youth) and adolescents, gave full informed consent to participate and signed a consent form to indicate this. Young people aged <16 years also required written parental consent to participate. Study data were anonymized before analysis, and all consent records were stored separately. Following the anonymization of interview and focus group transcripts, recordings and transcripts with person-identifiable information were deleted. Where applicable, participants were reimbursed for any transport costs associated with taking part and given a £20 (US $25.56) gift voucher per interview or focus group as a token of gratitude for their involvement.

Systematic Scoping Review: Identify and Define Stages

The PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines [ 22 ] were followed, and studies were included if they contained primary data on psychosocial coping strategies for sexual and gender minority youth, were conducted with adolescents (aged 10-19 years), and were published in English. The MEDLINE, Embase, and PsycINFO databases were searched. Search terms included a range of terms to capture a sexual and gender minority focus (eg, “gender minorit*” or “LGB*”) and a range of terms for psychosocial coping strategies (eg, “Coping*,” “adaptive,” and “resilience”). No date restrictions were applied, and the searches ran up to January 19, 2022. A descriptive approach to synthesizing the evidence, as recommended by Arksey and O’Malley [ 23 ], was used. The methods and findings of the scoping review have been published elsewhere [ 14 ]. The systematic scoping review ran in parallel to the focus groups with sexual and gender minority youth and interviews with adult experts and parents, which are reported in the following section.

Interviews and Focus Groups With Sexual and Gender Minority Youth, Adult Experts, and Parents: Identify and Define Stages

A total of 6 focus groups, each with between 3 and 10 sexual and gender minority youth participants, were conducted between November 2021 and February 2022. To reach and recruit participants in the applicable age range from the target communities, we worked with 3 organizations supporting LGBTQ+ youth to advertise the opportunity. Focus groups were run in conjunction with these organizations, with their staff also attending to help young people feel comfortable and supported. Staff also assisted in the process of obtaining informed consent from sexual and gender minority youth and, for those aged <16 years, from their parents or guardians. Due to COVID-19 restrictions, all focus groups were hosted via videoconference. The sessions were audio recorded and transcribed. Once accurate transcripts were approved (by MFGL or ANG) and fully anonymized, the focus group electronic audio recordings were deleted. Participants were all secondary school-aged, primarily between the ages of 12 and 20 years, with those aged ≤15 years in a separate focus group. In total, 4 participants aged ≤25 years took part in the focus groups with the older participants because they had special educational needs (eg, learning disabilities) and, as such, were still engaged in secondary-level education or training. Table 1 provides demographic information about sexual and gender minority youth focus group participants; 81% (29/36) of the sexual and gender minority youth were gender minority youth (ie, their gender identity was not the same as their sex as recorded at birth). Many participants (14/36, 39%) were bisexual or pansexual. Approximately 1 in 5 sexual and gender minority youth (7/36, 19%) were of dual heritage (eg, European and West African) or from a migrant background (eg, the other White participants who were not White British).

In parallel, 16 one-to-one interviews were conducted with adult experts based in England, with 6% (1/16) of the participants in Wales, including parents of sexual and gender minority youth, between October 2021 and January 2022. A total of 25% (4/16) of the adults held posts as commissioners of public health services relevant to sexual health and well-being, roles that included consideration of the needs of sexual and gender minority youth. In total, 25% (4/16) of the experts worked in frontline practitioner roles supporting the health and well-being of young people, including sexual and gender minority youth (eg, clinicians working in child and adolescent mental health services). In total, 25% (4/16) of the experts were community-based professionals, such as sexual and gender minority youth workers and policing staff focused on reducing the mistreatment of sexual and gender minority individuals. A total of 25% (4/16) of the adults were parents of a sexual and gender minority adolescent interested in better supporting sexual and gender minority youth. As with the sexual and gender minority youth focus groups, interviews were conducted using videoconference software and audio recorded, and transcripts of the interviews were produced. Once anonymized and approved as accurate (by MFGL or ANG), the electronic audio recordings were deleted.

a 6 focus groups in total with between 5 and 11 participants each; 44 participants in total (including 8 youth workers).

b This item was an open-ended question; as such, 3 gender minority youths wrote Male or Female (ie, Male and Female here does not necessarily equate to being cisgender and male or female).

c N/A: not applicable.

Appointment of Digital Developer (Preparation for Delivering the Concept, Create, and Use Stages)

In January 2022, a tender specification for a digital developer was created based on the outcomes at that time from the identify, define, and position work outlined previously. A range of commercial developers were notified of the tender, and after a competitive process involving an assessment of providers’ submissions and web-based interviews, Bluestep Solutions Limited (Bluestep for brevity) were appointed. They supported the research team in the task of translating the findings that emerged from the preceding evidence-gathering stages (ie, the scoping review, interviews, and focus groups) to content for the digital resource. Bluestep’s expertise resided in developing engaging and user-friendly content aligned with the research team’s evidence-informed approach. Co-design workshops with sexual and gender minority youth participants were conducted to refine the pilot content and improve its look and feel (described in the Findings of the Co-Design Workshops With Sexual and Gender Minority Youth: Position and Concept Stages section). Bluestep provided a map of the potential structure and parameters of the digital resource that could be developed within the available budget. The original budget was £41,000 (approximately US $50,000). Some savings were made in the project’s overall budget, and additional funds were also sourced through the OU, resulting in a final budget of nearly £50,000 (approximately US $61,000). To remain within budget, Bluestep indicated that the research team should focus on 3 core sections of content and have only 1 full day of filming.

Initial Co-Design Workshops With Sexual and Gender Minority Youth and Email or Web-Based Consultations: Position and Concept Stages

Two initial co-design workshops were held with (1) older sexual and gender minority youth aged ≥16 years (May 2022) and (2) younger sexual and gender minority youth aged 12 to 15 years (June 2022). The main aim of these workshops was to identify the priority issues and challenges faced by sexual and gender minority youth on which to focus and the preferred solutions and strategies that should be highlighted. To achieve this, 10 possible topics or issues and 11 possible solutions or strategies were presented to them based on data from the scoping review and the earlier interviews with adults and focus groups with sexual and gender minority youth. A modified nominal group technique [ 24 ] was used to facilitate this process. This involved structured voting before group discussions on the possible topics for inclusion, where all attendees were given an opportunity to express their views and preferences.

In June 2022, Bluestep created a selection of visual concepts ( Multimedia Appendix 1 ) with different color palettes and visual tones of voice represented by imagery. For example, the inclusive visualized toolkit included a bright rainbow color palette, and the message toning was intended to represent inclusivity and messaging that “we’re all in it together.” The overall concepts were also set out alongside some suggested names (created from a marketing perspective) from Bluestep for the digital resource. The suggested names, which drew on commercial marketing expertise from Bluestep, included the following:

• MEE: Mindful Education & Enlightenment for LGBTQ+
• Oneself: Defined by you, allied by us
• Free to be: Mindful tools for your journey

These visual concepts and suggested names were shared with our sexual and gender minority youth workshop participants, and they gave their feedback with support from youth workers via email and in a web-based consultation session via videoconference. The ultimate decisions about concepts, color schemes, and names were strongly informed by the sexual and gender minority youth’s views and we were led by their preferences. A set of questions to prompt discussions regarding preferences was provided to the youth workers supporting the consultation process.

Research Team Co-Development to Finalize Decisions on the Focus and Topic Areas (Define, Position, and Concept Stages)

Following the second sexual and gender minority youth co-design workshop, the research team met to reflect on the voting decisions of sexual and gender minority youth and discuss their own ideas for the priority content and sections in the resource and its features (eg, video clips and animations). In addition to professional expertise, members of the team also have lived experience from their personal lives on which to draw (eg, MFGL is a White migrant, queer male individual and gender role nonconformer; RS is from an ethnic minority group and has lived experience of mental illness [ 25 ]; and KB is White British, grew up with a sibling who identifies as a gay cisgender male, and has lived experience of mental illness). The team held 2 meetings 1 week apart in June 2022.

Questionnaire to Assess the Look and Feel of the Design Options: Concept Stage

Parallel to the co-design and development work outlined previously, Bluestep produced a number of design concepts for consideration by our sexual and gender minority youth workshop attendees and the research team ( Multimedia Appendix 1 ). A questionnaire was developed that asked sexual and gender minority youth workshop attendees to consider the designs and some other key features related to the look and feel of the resource, such as whether the characters featured should be real people or fully animated or whether the characters should be acting out scenarios versus sharing their own personal experiences as sexual and gender minority individuals ( Multimedia Appendix 2 ).

Appointment of Sexual and Gender Minority Contributors Through Specialist Media and Modeling Agencies: Concept Stage

On the basis of our understanding of the need for credible sources to deliver messages in our intervention, and because the dramatizations we had initially envisaged for Oneself in our original study protocol were deemed too contrived and artificial by sexual and gender minority youth, we made a notable decision. In particular, it was identified that real sexual and gender minority young adults, who can talk authentically about their own experiences growing up as sexual and gender minority individuals, would be an important feature of Oneself . In July 2022, the process of recruiting 3 sexual and gender minority young adult contributors or community members was initiated. We applied to modeling and talent agencies given that we wanted contributors who were comfortable in front of cameras. We were provided with a dozen portfolios of different potential sexual and gender minority contributors and short introductory video clips on why they were interested in being involved in the development of Oneself . The research team and sexual and gender minority youth considered the clips separately, and the sexual and gender minority youth voted on their preferred contributors or community members. Feedback on the initial possible contributors highlighted that there was a lack of diversity, particularly regarding ethnicity and body size (ie, they looked “too much like models”). In our attempts to ensure a broader representation, we went back a second time to the agencies to obtain further potential contributor options.

Further Co-Design Workshops With Sexual and Gender Minority Youth: Create Stage

In total, 2 additional co-design workshops were held in September 2022 and January 2023. Co-design workshops were hosted in person with MFGL, Bluestep, or ANG present. Audio recordings were transcribed, and once accurate transcripts were approved (by MFGL or ANG) and fully anonymized, the co-design workshop audio recordings were deleted. Participants were all secondary school-aged, primarily between the ages of 12 and 20 years, with those aged ≤15 years in a separate workshop. Demographic information about workshop participants is presented in Table 2 . A total of 93% (14/15) of the participants were gender minority youth (ie, their gender identity was not the same as their sex as recorded at birth), and 60% (9/15) were bisexual or pansexual. Approximately one-quarter of sexual and gender minority youth (4/15, 27%) were of a dual heritage (eg, Asian and Black) or from a migrant background (eg, White participants who were not White British). An in-person consultation also bridged co-design workshops 3 and 4. This was not recorded.

a 4 co-design workshops in total (with between 5 and 8 youth participants each); 19 participants in total (including 4 youth workers).

Results of the Systematic Scoping Review: Identify and Define Stages

The findings of the scoping review have been published previously [ 14 ]; however, a summary is presented in this section of what we learned that fed into our thinking about the content for Oneself . A total of 68 articles were identified as meeting the review criteria. The oldest paper dated from 2008, and more than half (25/68, 51%) were published from 2017 onward. Most studies (40/68, 59%) were small scale (ie, with <50 participants), and more than two-thirds (47/68, 69%) were conducted in the United States. In total, 26 studies included sexual minority youth only, a further 28 included sexual and gender minority young people, and 14 studies included only gender minority young people.

A total of 24 of the included articles focused on 17 unique interventions to support sexual and gender minority youth. More than half of the intervention papers (13/24, 54% studies) focused on both sexual and gender minority youth. In total, 9 studies included only sexual minority young people, and 2 studies focused on gender minority youth only. Of the 17 interventions, the most frequently cited therapeutic modality was cognitive behavioral therapy (11/24, 46% studies and 6/17, 35% interventions). Common features described in these interventions, including those with CBT-based modalities, are summarized in Table 3 [ 14 ].

Most of the interventions involved in-person delivery (14/24, 58% studies). In total, 5 (56%) out of 9 interventions were delivered in a digital format. In addition to the strategies and techniques outlined in Table 3 , it was also noted that interventions often sought to affirm sexual and gender minority youth identities and give a message of hope to intervention users (eg, “I won’t always feel this way” in the Rainbow SPARX intervention [ 26 ]).

A total of 44 of the included studies did not focus on interventions per se. Instead, they were mainly qualitative studies (with some mixed methods studies combining survey and qualitative data) that explored the experiences of sexual and gender minority youth and the strategies they used to cope with the challenges they face. Table 4 [ 14 ] summarizes the commonly identified strategies and tools for sexual and gender minority youth drawn from these studies and applied to Oneself .

Taken together, the strategies listed in Tables 3 and 4 gave us a comprehensive list of potential contenders to make up the core content and features of Oneself . We drew on this information and the findings we present in the following section from our focus groups and interviews to develop the content of the 3 topic areas identified as most important.

a CBT: cognitive behavioral therapy.

Results of Interviews and Focus Groups With Sexual and Gender Minority Youth, Adult Experts, and Parents: Identify and Define Stages

To expedite drawing out the relevant data from the focus group and interview transcripts and inform Oneself’s development, the data were divided between the research team and examined carefully. Detailed notes were made regarding the sorts of issues that the various stakeholders identified as important to address. Details on strategies and tools that were deemed useful in participants’ experiences were also extracted. The issues and strategies identified were revised during 2 team meetings in June 2022. A more detailed framework analysis [ 27 ] of the data is underway and will be published in due course.

The rapid data extraction process provided us with a series of initial issues and potential areas or populations of focus. MFGL and ANG then met to construct a long list of the main issues (n=10) and the potential solutions or strategies (n=11) that emerged from the findings of the scoping review and the interviews and focus groups with stakeholders. These are summarized in Textboxes 2 and 3 .

• How to deal with unsupportive parents or other family members
• How to deal with bullying at school (eg, name calling)
• How to deal with the challenges associated with coming out
• How to deal with negativity directed at lesbian, gay, bisexual, transgender, and queer people (eg, from a religion)
• How to deal with misgendering
• How to deal with feeling isolated or alone
• How to deal with stigma (eg, homo-, bi-, or transphobia)
• How to deal with web-based abuse (eg, trolls saying nasty things)
• How to explore and make sense of your sexuality or gender
• How to deal with people not believing you about your sexuality or gender
• Educate teachers and others on how to better support lesbian, gay, bisexual, transgender, and queer (LGBTQ+) youth so that school environments can be improved for LGBTQ+ youth
• Educate parents (and other people in the community) on how to better support LGBTQ+ youth so that communities can be improved for LGBTQ+ youth
• Help young people with practical issues—in particular, finding a toilet that they can safely use
• Allow the young person to connect directly with other LGBTQ+ young people so that they can talk to someone else who understands
• Coming out and how to do this safely—highlight that it is OK not to come out (and it is also OK to change one’s mind)
• Up-to-date and accurate information on sexuality and gender to help them make sense of their identity
• How to find supportive people via web-based environments so that they have a better support network
• Help young people figure out what they can and cannot change themselves so that they know what to focus their energy on
• Use affirmations (positive messages) about the young person (eg, “I deserve kindness” and “my gender is not an inconvenience”) to help people feel even better about themselves
• Help young people engage in creative activities (eg, art and music) to make them feel better
• Provide the contact phone numbers and details for supports available to LGBTQ+ youth so that they know where to go for extra help

First Proposed Structure and Designs After Appointing the Digital Developer: In Preparation for the Concept, Create, and Use Stages

Bluestep provided a map of the potential structure and parameters of the digital resource that it would be possible to develop within the available budget, specifically a wireframe. A copy of the structure is provided in Multimedia Appendix 3 . This illustrates the inclusion of 3 core features or sections of content and a “free” (all content is free to access, but the main content requires the user to create an account with a username or email address and password) taster section of content proposed as important to engage potential users and educate the wider public (eg, teachers).

Findings of the Co-Design Workshops With Sexual and Gender Minority Youth: Position and Concept Stages

Table 5 shows the average rank order preferences from the adapted nominal group technique voting in relation to priority issues or topics to cover within the Oneself resource. Participants ranked their highest-priority topic as rank 1 and their lowest-priority topic as rank 10. The lowest average rank order identifies the highest preference among the group. Dealing with unsupportive parents or other family members and dealing with bullying were the highest-ranked topics to cover. Table 6 presents the average rank order preferences for possible solutions or strategies to include in Oneself . The highest-ranking content included educating parents and teachers to help improve the quality of the environments they live in.

a LGBTQ+: lesbian, gay, bisexual, transgender, and queer.

Findings of the Research Team’s Co-Design Meetings in June 2022: Design, Position, and Concept Stages

The first co-design meeting with the research team began by reflecting on the rank order preferences of the sexual and gender minority youth (presented previously). It was acknowledged that, although clear priorities emerged from the data, there was also considerable variability in the rank order preferences. With the budget and practical limits to the amount of content that we could include, we could not create an ideal resource to suit all sexual and gender minority youth needs. However, given the identification by Bluestep that we could have three main sections with featured content, the selection of the top three topic areas was straightforward: (1) coming out and doing so safely; (2) managing school, including homophobic, biphobic, or transphobic bullying or similar; and (3) dealing with parents and families, especially unsupportive family members, including parents or caregivers. We found the favored focus area or population being about educating parents, teachers, and other community members to be outside the scope given our budget to date and as the resource was always intended to be primarily for sexual and gender minority youth themselves rather than adults who support them. The resource is designed to center the experiences of sexual and gender minority youth, but we expect that Oneself will ultimately support parents, teachers, and other community members by increasing awareness and visibility of sexual and gender minority youth experiences. We do acknowledge that there are important challenges in balancing (individual-focused) support for sexual and gender minority youth with promoting social justice through education of adult stakeholders. As was done in this study, it is important to consider these elements in parallel because they are interactive. While we decided not to explicitly target adults at this stage, we acknowledged this request as being part of sexual and gender minority youth’s desire for the environments they live in to be better and more supportive of them, hence the decision to prioritize the educational animation about pronouns, intended for a wider audience (including parents and teachers). We also reflected on the fact that, while the main purpose of the content should be to help young people cope with situations independently, it could also be useful for educating parents, teachers, and other members of the community. Specifically, the resource could help them understand the unique challenges of growing up as a sexual and gender minority youth and how they can act and respond supportively to promote positive social change. At this stage, we thought that the formats we might use to present content could be videos or animations depicting narratives of sexual and gender minority youth everyday experiences, possibly with some interactive content or features for the user.

Sexual and gender minority youth understandably had a range of perspectives and ideas about what should be covered in Oneself . We identified 9 such specific suggestions. For instance, we were cautioned against educating Oneself users on the various sexuality and gender “labels” used by a young person given that the terminology is continually evolving (and frequently contested). Another sexual and gender minority youth felt strongly that we should acknowledge the difficulties associated with challenging environments; for example, “you cannot change everyone,” and therefore, a sexual and gender minority youth must know how (and when) to “walk away.” They also wanted us to ensure that our sexual and gender minority contributors would represent as much diversity as possible. By the end of the research team discussions, there was a growing sense that we could cover, to some degree, many of the preferred solutions or strategies that had been discussed and voted on by sexual and gender minority youth in their co-design workshops, with a focus on the top 3 topics or issues.

It was beyond the scope and resources of Oneself to provide a web-based community space where sexual and gender minority youth could connect with each other safely in real time as this would likely require constant monitoring and ongoing administration. However, advice on where or how to do this elsewhere could be included, along with links to other supportive resources. It was decided that the resource would focus on supporting sexual and gender minority youth directly. We aimed to center the young person in this resource, with Oneself often talking directly to them and trying to focus on them and their needs, for instance, by using language or terms and concepts that map to the concerns they have raised with us as the research team. This act of centering is in direct contrast to the marginalization that they may face daily. It was also intended to have a dual purpose of potentially serving to educate the wider community, including parents and teachers. It was felt that, because the 3 main topics focused on dealing with challenges that can have a detrimental effect on well-being, the resource needed to include evidence-based tools and resources known to support and enhance mental well-being, such as relaxation techniques and other relevant means of coping. It also needed to include content that felt empowering of developing and evolving identities to support and develop users’ self-esteem.

Findings of the Questionnaire to Assess the “Look and Feel” Design Options: Concept Stage

The wireframe structure of Oneself (which was designed to include some introductory content) was confirmed first. This included a log-in feature to access the 3 main content sections and recommended additional resources and sources of help and support. The log-in feature, with the associated gathering of demographic data, was deemed necessary to capture future user information related to Oneself. Next, Bluestep worked with the research team to develop a questionnaire posing different design concepts and options for the look and feel of the resource. The full questionnaire and the options posed are presented in Multimedia Appendix 2 . The preferences that this process helped identify are briefly summarized in the following paragraph.

Although the idea for having full animations with voice actors was rated favorably by many sexual and gender minority youth participants, a clear overall preference emerged for using real people talking about their firsthand experiences growing up as sexual and gender minority youth, as well as the inclusion of sexual and gender minority youth “influencers” or public figures. There were also clear indications that the resource would most likely be accessed on a smartphone by sexual and gender minority youth and that video clips should include audio subtitles (to enable viewing without sound on; however, this is also valuable for accessibility reasons), and most indicated that they would use headphones to listen to content, too. On the basis of sexual and gender minority youth feedback, video-based content should ideally not exceed 60 seconds; some were willing to watch longer clips when the content was engaging. Downloadable information sheets, for access again offline, were identified as useful, and sexual and gender minority youth participants favored a color palette that was pastel and informed by the “progress rainbow flag.”

Team Consultation Based on the Questionnaire Feedback Led to Plans for Inclusion of Sexual and Gender Minority Contributors: Concept and Create Stages

The feedback we obtained about the inclusion of sexual and gender minority contributors (ie, not actors playing a role) led to further consultation about the format of the resource and a decision to focus the main content on testimonial or account footage from sexual and gender minority young adults who could reflect on their experiences with the topics selected when they were growing up. We set out to identify individuals from modeling and talent agencies who would be willing to provide this kind of content, as described in the Appointment of Sexual and Gender Minority Contributors Through Specialist Media and Modeling Agencies: Concept Stage section.

The process of assessing potential sexual and gender minority content contributors resulted in the appointment of 3 people who identified as sexual and gender minority individuals who were willing to be involved for a set fee. Between them they represented diversity in terms of gender and sexual identity, body shape and size, ability, and ethnicity. More details about those selected are provided in the following section.

Design Concept Selection via Email and Web-Based Consultation With Sexual and Gender Minority Youth and Outcomes From Co-Design Workshops 3 and 4: Position, Concept, and Create Stages

Concept 1 ( Figure 1 ) was a clear favorite in terms of the color scheme, and it was described as more “friendly” and inclusive than concept 2 ( Figure 2 ). There was a question regarding the icons in both concepts (ie, symbols transposed over certain images); sexual and gender minority youth did not feel that the icons represented the topics adequately, and therefore, wording or text would be needed, which would defeat the purpose of using icons. In concept 1, a “share” function was seen as more understandable as it was interpreted as a speech bubble, though this could be made even clearer.

From concept 2, sexual and gender minority youth liked the “squiggly lines” in the designs if they could be incorporated into concept 1’s color scheme. It was preferred that design elements from both concepts could be used in the final resource, although sexual and gender minority youth were clear not at the same time as it would be too much on one image.

The sexual and gender minority youth participants were asked if they thought that including the OU (lead university for the project) logo on the resource was a good idea. Most participants felt that it would give people confidence in the quality of the resource as OU is a well-known brand in the United Kingdom. The preferred name for the resource, of the 3 suggestions, was Oneself , but they considered the inclusion of the originally proposed taglines to be too long. Consequently, we did not use a subsequent lengthy tagline in combination with the name Oneself across the whole resource.

Table 7 provides a summary of the workshops and consultations by date, including what was covered and how it aligns with the co-design stages by Hagen et al [ 13 ].

a OU: The Open University.

Introducing the Sexual and Gender Minority Contributors: Create Stage

Bluestep shortlisted 10 candidate sexual and gender minority young adult contributors for the research team, who in turn shortlisted 5 to present to the young people in co-design workshop 3. There were some unforeseen recruitment difficulties. For example, the selected racial and ethnic minority gay man and a transgender woman (who was one of the sexual and gender minority youth’s top choices) were unfortunately not able to participate as initially agreed. For instance, one of them became concerned about how publicly accessible Oneself would be once released (ie, they could be “outed” to a whole range of people known to them). Thus, 2 female contributors were selected from the initial shortlist, and both were rated very favorably by the sexual and gender minority youth. As it was important for the project to reflect diversity across gender identity, sexuality, race, and disability, a further search for a third contributor was carried out in October 2022. Finally, 3 contributors were selected and approved by the young people: Chloe, Lilly, and Georgie.

Georgie, also known as Triple Minor, uses they, she, or he pronouns and is transgender nonbinary. Georgie wanted to contribute to Oneself because they were keen to be the much needed representation that is often lacking within LGBTQ+ communities.

Lilly uses she or her pronouns and is pansexual. Lilly wanted to contribute to Oneself because, when she was younger, she would have loved to have heard more about queer perspectives. This is why she wanted to talk about her own experiences.

Chloe uses she or her pronouns and is a lesbian. Chloe wanted to contribute to Oneself because she believes it is important for the younger LGBTQ+ community to feel supported and comfortable in their sexuality and be able to hear the voices and perspectives of queer people.

Bluestep developed and circulated a creative brief for the 3 sexual and gender minority contributors explaining the requirements for filming ( Figures 3 and 4 ).

Filming took place on November 29, 2022, in a London-based studio. On the day, all 3 sexual and gender minority contributors were asked the same questions on the topics of school, coming out, and friends and family ( Multimedia Appendix 4 ). Filming was done against a green screen so that animations could be added later. Rough-cut footage included approximately 35 minutes of Chloe, Lilly, and Georgie each and 10 minutes of a group recording. ANG transcribed these rough cuts, which comprised 28 pages in total, and summarized their content into key points and quotes that could be shared with the young people. These were given to the sexual and gender minority youth in co-design workshop 4, who rated the points and quotes, adding their own reflections. For instance, the sexual and gender minority youth found Lilly’s advice to cope if someone reacts negatively to coming out helpful—she said the following: “Remember you are not alone. It may take time, but you’ll find your community and people that get you and understand you.” However, the sexual and gender minority youth found Georgie’s advice for teachers and students to manage bullying at school (ie, “zero tolerance” for this) too vague as most schools should have zero tolerance policies but there is still a need for proactiveness to enforce them. A summary of these points, organized by topic (coming out, school, and family and friends) and divided into challenges and solutions and strategies with key quotes to include, was then given to Bluestep to create 2- to 3-minute–long rough cuts of each video, which combined live footage and animation. This resulted in a total of 6 videos— Parents and Families: Some Common Challenges , Parents and Families: Some Strategies , School: Some Common Challenges , School: Some Strategies , Coming Out: Some Common Challenges , and Coming Out: Some Strategies . These were reviewed several times for content, design, storyline, accessibility, and subtitles and finally refined before approval by MFGL and ANG.

The Oneself Resource

Oneself was divided into 7 web pages: a home page ( Figure 5 ); the 3 topics of parents and families, school, and coming out; downloads; chilling out; and resources. To access the entire toolkit, the user needs to log in and complete a brief baseline measure of well-being (ie, the 5-item World Health Organization Well-Being Index). The home page is free to access for anyone, although images of the contributors are reserved for the logged-in user.

The home page includes a description of Oneself ; quotes; and extracts of what the user could find in the resource, for instance, the 3 topics. These were designed to prompt the user to log in to access the content. The home page also features an animation of the meaning and use of pronouns created in collaboration with sexual and gender minority youth from Rainbow Power (a sexual and gender minority youth group) run by the Free2B Alliance in England.

Each topic area began with two parts: (1) the problems and challenges that sexual and gender minority youth face in relation to that topic and (2) potential strategies and solutions to these issues. Each topic area included videos and social polls, which were then followed by activities, downloadable exercises, and external resources (see Multimedia Appendix 5 for an example).

Each topic area had 2 live footage videos: the first with sexual and gender minority contributors talking about common challenges on the topic based on their own experiences and the second with sexual and gender minority contributors talking about solutions, strategies, and advice on the topic based on their own experiences. Live footage was mixed with an animated background, highlighting what contributors were speaking about with color, illustrations, or additional text.

Each topic area also had 2 social polls, the first of which asked users to reflect on their own experiences on the topic. For example, for Coming Out , the question was as follows: “have you come out to others about your sexuality or gender yet?” A second social poll question asked users to reflect on which contributor’s experience was most similar to their own. After answering, the percentage of responses to the question became visible to the user. This was designed so that the user could understand others’ experiences and feel part of the Oneself community.

In total, 2 exercises or activities per topic area were designed to help the user reflect on the topic in greater depth and learn more about how to implement strategies and advice in managing challenges. For instance, an exercise in “Parents and Families” is framed as follows: “Some LGBTQ+ young people have repeatedly described online environments as ‘lifesaving’ at times. Reflect on your experiences of creating an online support network for yourself.” This is followed by an “Explore More” button that takes the user to another page where they can read through several strategies and choose the ones that fit them best ( Figure 6 ).

Downloads or downloadable exercises for each topic area were drafted by MFGL and ANG and then checked and refined by other research team members. Downloads were designed to tackle the problems and challenges described in each of the topics—2 downloadable guides addressing issues relevant to each topic provide detailed written information on strategies and solutions regarding them. In the case of Parents and Families , these look at standing up for yourself (communication) and problem-solving. In the case of School , they focus on finding allies and rejecting negativity (ie, the ABCD method). In the case of Coming Out , they support the coming out journey and finding hope. Downloads can also be found grouped together under the Downloads tab. An overview of the logic underpinning the development and content of Oneself is depicted in Figure 7 .

Finally, each topic area provided 2 external resources leading to organizations and web pages that can offer sexual and gender minority youth further support, such as advice, community resources, or helplines. These were chosen in agreement with the research team. External resources can also be found grouped together under the Resources tab.

All content was interspersed with quotes from our sexual and gender minority youth participants and the 3 sexual and gender minority contributors as well as short comments and advice linked to the social polls.

Finally, a Chilling Out section was included promoting relaxation exercises as well as 2 additional external resources. These exercises consisted of 3 recordings, each led by a sexual and gender minority contributor following a script provided by the research team ( Figure 8 ). A stress levels scale of 1 to 10 was available to complete before and after listening to each recording to help the user reflect on whether it had been a useful and calming exercise for them. Oneself was designed so that users can rate content with between 1 and 5 stars as they work through it, providing the research team with feedback.

Principal Findings

This study aimed to set out the systematic and iterative approach undertaken to develop a web-based resource to support the mental well-being of sexual and gender minority youth so that they can deal effectively with the specific challenges of growing up LGBTQ+. Providing this kind of support was identified as important because sexual and gender minority youth are at greater risk of poor mental health outcomes than their cisgender and heterosexual peers [ 1 , 4 , 6 , 14 ]. This study demonstrated how project activities are mapped against the 6 stages of co-design set out by Hagen et al [ 13 ]. In particular, it showed how extant research evidence and engagement with a range of stakeholders and representatives of end users were drawn on to make decisions about the content and design of the final resource, named Oneself . The logic underpinning the content of the resource is also set out to support the design of future process and outcome evaluations of Oneself . Initial usability and end-user feedback has been gathered through a set of “think aloud” interviews and post use reflection interviews with sexual and gender minority youth and adult stakeholders. The findings of the latter study will be reported in detail elsewhere (MFG Lucassen, unpublished data, 2024). The feedback to date has been largely positive, with all sexual and gender minority youth testers saying they would recommend the resource to others. There have also been some points of constructive and critical feedback, in particular from adult stakeholders, which will need to be considered in future development work (eg, that a greater range of experiences should be included, such as those of cisgender gay male individuals and individuals from minority faith backgrounds).

Strengths and Limitations of Oneself

There are several strengths and limitations to Oneself in its current format. Strengths include the fact that Oneself represents one of the first digital mental well-being–related resources reported on that is designed specifically to meet the needs of sexual and gender minority youth. The design drew on the evidence base for techniques to support their well-being [ 14 ] and was developed in collaboration with 4 different stakeholder groups: sexual and gender minority youth, adults who work with sexual and gender minority youth, parents of sexual and gender minority youth, and commissioners of public health services focused on their needs. In doing so, the process of development included a wide variety of relevant perspectives and looked to build on what is already known about supporting the well-being of this population. It is also a strength of the resource that its development included adolescents aged <16 years and was inclusive of gender and sexual minority groups rather than focusing solely on gender or sexual minority groups. This is a departure from previous interventions that have typically focused on those aged >16 years only and selected to focus on either gender or sexual minority groups [ 14 ]. Although there are important differences between sexual and gender minority experiences, there is also considerable overlap, including compromised mental well-being for many. Some young people will ultimately identify as being both a sexual and gender minority, which makes the resource’s recognition of both groups important.

Limitations of the resource include the fact that, given the budget constraints, much of the available funding had to be channeled into creating basic initial functionality that would be likely to engage and sustain interest from the target end users. This meant that much of the evidence-informed content that we might expect to have the greatest effect on mental health and well-being had to be included within the more text-heavy “downloads” section. Although, in early consultation work, sexual and gender minority youth suggested that these “downloads” were a good way to provide additional resources for use offline, it was later acknowledged that young people do not want to have to read a lot of text when engaging with the content (MFG Lucassen, unpublished data, 2024). Common evidence-based features for supporting mental health and well-being include relaxation exercises [ 28 ], behavioral activation [ 29 ], problem-solving [ 28 - 30 ], helping people recognize problematic cognitions [ 26 ], and cognitive restructuring [ 26 ]. Future iterations will need to focus on bringing more of this content into the interactive elements of Oneself. However, in doing so, it will also be important to consider whether such features are best delivered via pure self-help or whether optimal delivery requires engagement with an adult who can help structure what are often quite complex therapeutic activities (eg, sexual and gender minority youth can feasibly be supported by “e-coaches” to complete resources such as Oneself ).

The sexual and gender minority youth involved in co-designing Oneself included almost one-quarter of individuals who were of dual heritage (eg, European and West African) or from a migrant background (eg, several of the White participants). Furthermore, gender minority youth, who have been traditionally underrepresented in LGBTQ+ research [ 31 ], were very well represented, as were bisexual and pansexual participants. Nonetheless, content could have been improved in relation to intersectionality, such as the fact that there is a need to represent sexual and gender minority youth more complexly in terms of sexual and gender minority youth’s social positions (eg, across ethnicity, religion, and social class). Future iterations need to look at making the resource more relatable to additional underrepresented groups, as was suggested during co-design processes (eg, for care-experienced Asian sexual and gender minority youth from a minority faith background), who may face different and complex challenges growing up as sexual and gender minority youth.

Self-help digital resources and interventions have the potential to be very cost-effective [ 32 ]. They can be relatively low cost to produce, with the potential for very high reach given evidence of increasing digital access and capability, particularly among young people [ 10 , 11 ]. Despite this, it is likely that those with the greatest vulnerabilities and at the most risk of poor mental well-being may be the least likely to access suitable web-based spaces with ease (eg, those with limited funds to purchase data for a mobile phone). Therefore, reaching these individuals needs to be carefully considered by those who are responsible for identifying and tackling such needs, including youth support organizations, schools, and commissioners of services. In addition, digital resources such as Oneself need to keep up with the rapid pace of progress and evolution in the web-based world. Young people have high expectations and are savvy consumers of web-based media, and they anticipate polished and engaging products. Keeping a resource such as Oneself comprehensive, up-to-date, and relevant in terms of content and look and feel requires ongoing funding. Relatedly, sexual and gender minority youth highlighted the importance of educating others, in particular teachers and parents, as this would bolster their overall mental well-being. In future funded work, we would like to develop resources specifically for adults, potentially within the overall Oneself intervention. Finally, something we identified that we would not be able to achieve with Oneself , at least for now, was direct access to support and interaction from a sexual and gender minority youth peer group. Although this was desired, providing it would involve considerable resources to monitor and approve content and messaging and avoid harm that could be caused by web-based bullying and harassment. Investigating how to provide this sense of community more fully in a web-based space warrants further attention. Ideally, such spaces should be structured in such a way that the experiences of sexual and gender minority youth can be shared without any pressure to divulge information that could identify a young person or lead to instances of “oversharing” (which sexual and gender minority youth may regret at a later stage). Case studies, as presented in Oneself with the contributors, could offer a safe means to discuss personal issues without the need for self-disclosure. We think that establishing and maintaining web-based community spaces in the context of digital mental health technologies requires further study to ensure that such spaces are both acceptable and viable. However, a noteworthy shortcoming of direct access to ongoing human support and interaction given the associated costs and practical considerations (eg, whether an intervention can realistically be provided 24 hours a day, 7 days a week) are limitations in terms of an intervention’s likely reach.

Strengths and Limitations of the Research

The research we have conducted in developing Oneself, and this paper specifically, makes an important contribution to needed literature that opens the “black box” of intervention development [ 33 ]. Attempting to record the process of development, including the co-design stages, as accurately and comprehensively as possible and placing it within the public domain via open access publishing contributes to the Open Science Agenda by making it accessible, inclusive, and transparent [ 34 ]. Being explicit about the logic that underpins the intervention content in terms of how it is intended to have an effect on factors associated with maintenance (or not) of mental well-being is also important to support the design of future evaluation studies [ 35 ].

Co-design work is complex and challenging to do well. We believe that aspects of our co-design efforts were of merit, in particular our inclusion of younger sexual and gender minority youth (which included participants as young as 12 years of age) and our engagement of sexual and gender minority youth from the “Identify” all the way through to the “Use” stages of the process [ 13 ]. We drew heavily on sexual and gender minority youth’s views to decide on the topic areas to focus on and in deciding on the “look and feel” of Oneself . We also made key changes to the resource in response to sexual and gender minority youth feedback, such as not using dramatizations, as was initially envisaged. Challenges to the co-design processes included the COVID-19 pandemic at the start of the project, which meant that work with sexual and gender minority youth was conducted using videoconference software at a time when adolescents were frequently fatigued by web-based forms of communication. Connected to this was our awareness that assisting in the creation of Oneself was one of the many demands placed on the sexual and gender minority youth involved in co-design, and as such, we sought to use sexual and gender minority youth’s time efficiently. Consequently, we limited the number of workshops conducted and carried out some consultations via email, which was less robust. In the future, we could enhance our co-design efforts and move closer to partnership (as opposed to consultation as defined by Arnstein [ 36 ] in her ladder of participation) by helping a number of older sexual and gender minority youth learn more about evidence-based techniques for supporting mental well-being and subsequently getting them to design features of the content. These older adolescents could be employed as coresearchers, and they could draft and further develop content with our ongoing support.

Most sexual and gender minority youth involved in the co-design of Oneself were gender minority young people, which is a strength given that these youth are underserved by mental health services [ 4 ]. However, a limitation of our research was that we struggled to recruit cisgender adolescents to join the co-design workshops and, as such, may have underrepresented the views or specific needs of certain youth (eg, cisgender lesbian and gay youth). Relatedly, it is likely that some groups or individuals who may need intervention support the most are among those least likely to get involved in co-design or research activities (eg, sexual and gender minority youth who do not feel safe to “come out”) leading to intervention development and associated research more generally, which misses the perspective of those who are “not out.” Acknowledging this potential issue is important, and striving to reach the underheard and underserved must remain a priority of future research.

Summary, Conclusions, and Next Steps

This study aimed to set out the process involved in co-designing and developing Oneself , a digital resource to support sexual and gender minority youth in building and maintaining their resilience to cope with the everyday challenges of growing up LGBTQ+ and support their mental health and well-being more generally. It is hoped that, in the future, this resource will be extended so that it is also of use for educating adults who wish to support sexual and gender minority youth. We have explained the included content and the logic that underpins its use and acknowledged a range of strengths and limitations of what has been achieved so far. Priorities for future efforts will be to specifically address critique and feedback provided by adults and sexual and gender minority youth during their “think aloud” interviews (MFG Lucassen, unpublished data, 2024); build in additional characteristics translating evidence-based content into interactive features; and continue to incorporate diverse voices in co-design, including consideration of how intersectionality may need to be more integrated. The next steps include applying for further research funding to continue our evaluation and development activities.

Acknowledgments

The authors would like to thank all the study participants, academic advisors, and organizational partners for this project, such as the lesbian, gay, bisexual, transgender, and queer organizations, specifically Free2B Alliance and METRO Charity. The authors also thank their other partners, which include 2 county councils, a clinical commissioning group, and the Centre for Policing Research and Learning (at The Open University). The authors would also like to thank Lauren Walker for her feedback on earlier drafts of this paper. Funding for this project was provided by the UK Medical Research Council (grant MR/V031449/1).

Data Availability

The data sets generated during and analyzed during this study are not publicly available due to their sensitive nature but are available from the corresponding author on reasonable request.

Conflicts of Interest

The authors are the codevelopers of Oneself but do not stand to gain financially from its future use and have no further conflicts of interest to declare.

Concept design options for Oneself.

Questionnaire for sexual and gender minority youth as part of co-design of Oneself.

Wireframe website map for Oneself.

Instructions given to sexual and gender minority contributors for filming video content.

Screenshots from Oneself.

• Lucassen MF, Stasiak K, Samra R, Frampton CM, Merry SN. Sexual minority youth and depressive symptoms or depressive disorder: a systematic review and meta-analysis of population-based studies. Aust N Z J Psychiatry. Aug 2017;51(8):774-787. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Reisner SL, Poteat T, Keatley J, Cabral M, Mothopeng T, Dunham E, et al. Global health burden and needs of transgender populations: a review. Lancet. Jul 23, 2016;388(10042):412-436. [ FREE Full text ] [ CrossRef ] [ Medline ]
• LGBT+ Pride 2021 global survey points to a generation gap around gender identity and sexual attraction. Ipsos. 2021. URL: https:/​/www.​ipsos.com/​en/​lgbt-pride-2021-global-survey-points-generation-gap-around-gender-identity-and-sexual-attraction [accessed 2024-03-13]
• Clark TC, Lucassen MF, Bullen P, Denny SJ, Fleming TM, Robinson EM, et al. The health and well-being of transgender high school students: results from the New Zealand adolescent health survey (Youth'12). J Adolesc Health. Jul 2014;55(1):93-99. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Meyer IH. Prejudice, social stress, and mental health in lesbian, gay, and bisexual populations: conceptual issues and research evidence. Psychol Bull. Sep 2003;129(5):674-697. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lucassen MF, Samra R, Rimes KA, Brown KE, Wallace LM. Promoting resilience and well-being through co-design (the PRIDE project): protocol for the development and preliminary evaluation of a prototype resilience-based intervention for sexual and gender minority youth. JMIR Res Protoc. Feb 01, 2022;11(2):e31036. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lucassen MF, Clark TC, Denny SJ, Fleming TM, Rossen FV, Sheridan J, et al. What has changed from 2001 to 2012 for sexual minority youth in New Zealand? J Paediatr Child Health. Apr 10, 2015;51(4):410-418. [ CrossRef ] [ Medline ]
• Gnan GH, Rahman Q, Ussher G, Baker D, West E, Rimes KA. General and LGBTQ-specific factors associated with mental health and suicide risk among LGBTQ students. J Youth Stud. Feb 17, 2019;22(10):1393-1408. [ CrossRef ]
• Evans C, Robertson W. The four phases of the digital natives debate. Human Behav and Emerg Tech. Jun 17, 2020;2(3):269-277. [ FREE Full text ] [ CrossRef ]
• Petrosyan A. Internet usage in the United Kingdom - statistics and facts. Statista. URL: https://www.statista.com/topics/3246/internet-usage-in-the-uk/#topicOverview [accessed 2024-03-11]
• Children and parents: media use and attitudes report 2022. Ofcom. 2022. URL: https:/​/www.​ofcom.org.uk/​__data/​assets/​pdf_file/​0024/​234609/​childrens-media-use-and-attitudes-report-2022.​pdf [accessed 2024-03-11]
• McDermott E, Hughes E, Rawlings VE. Queer futures: understanding lesbian, gay, bisexual and trans (LGBT) adolescents’ suicide, self-harm and help-seeking behaviour. Department of Health Policy Research Programme Project. 2018. URL: https://www.queerfutures.co.uk/wp-content/uploads/2016/06/Queer-Futures-Final-Report.pdf [accessed 2024-03-23]
• Hagen P, Collin P, Metcalf A, Nicholas M, Rahilly K, Swainston N. Participatory design of evidence-based online youth mental health promotion, prevention, early intervention and treatment. Young and Well Cooperative Research Centre. 2012. URL: https:/​/www.​westernsydney.edu.au/​__data/​assets/​pdf_file/​0005/​476330/​Young_and_Well_CRC_IM_PD_Guide.​pdf [accessed 2024-03-11]
• Lucassen MF, Núñez-García A, Rimes KA, Wallace LM, Brown KE, Samra R. Coping strategies to enhance the mental wellbeing of sexual and gender minority youths: a scoping review. Int J Environ Res Public Health. Jul 18, 2022;19(14):8738. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jagosh J, Macaulay AC, Pluye P, Salsberg J, Bush PL, Henderson J, et al. Uncovering the benefits of participatory research: implications of a realist review for health research and practice. Milbank Q. Jun 18, 2012;90(2):311-346. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Donetto S, Pierri P, Tsianakas V, Robert G. Experience-based co-design and healthcare improvement: realizing participatory design in the public sector. Des J. May 07, 2015;18(2):227-248. [ CrossRef ]
• Smith B, Williams O, Bone L, The Moving Social Work Co-production Collective. Co-production: a resource to guide co-producing research in the sport, exercise, and health sciences. Qual Res Sport Exerc Health. Mar 19, 2022;15(2):159-187. [ CrossRef ]
• Walker L, Dawson S, Brady S, Hillison E, Horspool M, Jones G, et al. Co-producing a physical activity intervention with and for people with severe mental ill health – the spaces story. Qual Res Sport Exerc Health. Jan 08, 2023;15(2):235-247. [ CrossRef ]
• Beresford P. From ‘other’ to involved: user involvement in research: an emerging paradigm. Nord Soc Work Res. Nov 2013;3(2):139-148. [ CrossRef ]
• Bevan Jones R, Stallard P, Agha SS, Rice S, Werner-Seidler A, Stasiak K, et al. Practitioner review: co-design of digital mental health technologies with children and young people. J Child Psychol Psychiatry. Aug 22, 2020;61(8):928-940. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Michie S, Abraham C, Eccles MP, Francis JJ, Hardeman W, Johnston M. Strengthening evaluation and implementation by specifying components of behaviour change interventions: a study protocol. Implement Sci. Feb 07, 2011;6:10. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. Oct 02, 2018;169(7):467-473. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. Feb 2005;8(1):19-32. [ CrossRef ]
• Gallagher M, Hares T, Spencer J, Bradshaw C, Webb I. The nominal group technique: a research tool for general practice? Fam Pract. Mar 1993;10(1):76-81. [ CrossRef ] [ Medline ]
• Samra R. Beyond epistemic injustice: when perceived realities conflict. Harv Rev Psychiatry. 2023;31(5):223-225. [ CrossRef ] [ Medline ]
• Lucassen MF, Merry SN, Hatcher S, Frampton CM. Rainbow SPARX: a novel approach to addressing depression in sexual minority youth. Cogn Behav Pract. May 2015;22(2):203-216. [ CrossRef ]
• Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. Sep 18, 2013;13:117. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Heck NC. The potential to promote resilience: piloting a minority stress-informed, GSA-based, mental health promotion program for LGBTQ youth. Psychol Sex Orientat Gend Divers. Sep 2015;2(3):225-231. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Duarté-Vélez Y, Bernal G, Bonilla K. Culturally adapted cognitive-behavior therapy: integrating sexual, spiritual, and family identities in an evidence-based treatment of a depressed Latino adolescent. J Clin Psychol. Aug 21, 2010;66(8):895-906. [ CrossRef ] [ Medline ]
• Diamond GM, Diamond GS, Levy S, Closs C, Ladipo T, Siqueland L. Attachment-based family therapy for suicidal lesbian, gay, and bisexual adolescents: a treatment development study and open trial with preliminary findings. Psychotherapy (Chic). Mar 2012;49(1):62-71. [ CrossRef ] [ Medline ]
• Lucassen MF, Burford J. Educating for diversity: an evaluation of a sexuality diversity workshop to address secondary school bullying. Australas Psychiatry. Oct 30, 2015;23(5):544-549. [ CrossRef ] [ Medline ]
• Gentili A, Failla G, Melnyk A, Puleo V, Tanna GL, Ricciardi W, et al. The cost-effectiveness of digital health interventions: a systematic review of the literature. Front Public Health. 2022;10:787135. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Murray J, Baxter R, Lawton R, Hardicre N, Shannon R, Langley J, et al. Unpacking the Cinderella black box of complex intervention development through the Partners at Care Transitions (PACT) programme of research. Health Expect. Aug 2023;26(4):1478-1490. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Open science. United Nations Educational, Scientific and Cultural Organization. URL: https://www.unesco.org/en/open-science?hub=686 [accessed 2024-03-11]
• The Magenta Book: HM treasury guidance on what to consider when designing an evaluation. HMTreasury, Government of UK. 2020. URL: https://www.gov.uk/government/publications/the-magenta-book [accessed 2024-03-11]
• Arnstein SR. A ladder of citizen participation. J Am Inst Plann. Jul 1969;35(4):216-224. [ CrossRef ]

Abbreviations

Edited by A Mavragani; submitted 20.11.23; peer-reviewed by E Layland; comments to author 07.02.24; revised version received 23.02.24; accepted 26.02.24; published 21.05.24.

©Katherine Brown, Mathijs F G Lucassen, Alicia Núñez-García, Katharine A Rimes, Louise M Wallace, Rajvinder Samra. Originally published in JMIR Formative Research (https://formative.jmir.org), 21.05.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.