what is assignment bias

Definition of Assignment Bias

Assignment bias refers to a type of bias that occurs in research or experimental studies when the assignment of participants to different groups or conditions is not randomized or is influenced by external factors.

Understanding Assignment Bias

Randomized assignment or allocation of participants to different groups is a fundamental principle in research studies that aims to eliminate assignment bias.

Causes of Assignment Bias

Assignment bias can arise due to several reasons:

• Non-randomized allocation: When participants are not randomly assigned to different groups, their characteristics may influence the assignment, introducing bias into the study. This can occur when researchers purposefully assign participants based on certain characteristics or when participants self-select into a specific group.
• External factors: Factors external to the research design, such as the preferences of researchers or unequal distribution of participants based on certain characteristics, may unintentionally affect the assignment process.
• Selection bias: If participants are not selected randomly from the population under study, the assignment process can be biased, impacting the validity and generalizability of the results.

Effects of Assignment Bias

Assignment bias can have various consequences:

• Inaccurate estimation: The inclusion of biased assignment methods can lead to inaccurate estimations of treatment effects, making it difficult to draw reliable conclusions from the study.
• Reduced internal validity: Assignment bias threatens the internal validity of a study because it hampers the ability to establish a causal relationship between the independent variable and the observed outcomes.
• Compromised generalizability: The presence of assignment bias may limit the generalizability of research findings to a larger population, as the biased assignment may not appropriately represent the target population.

Strategies to Minimize Assignment Bias

To minimize assignment bias, researchers can undertake the following strategies:

• Randomization: Random allocation of participants to different groups reduces the likelihood of assignment bias by ensuring that each participant has an equal chance of being assigned to any group.
• Blinding: Adopting blind procedures, such as single-blind or double-blind designs, helps prevent the influence of researcher or participant bias on the assignment process.
• Stratification: Stratifying participants based on certain important variables prior to assignment can ensure a balance of these variables across different groups and minimize the impact of confounding factors.

• Bipolar Disorder
• Therapy Center
• When To See a Therapist
• Types of Therapy
• Best Online Therapy
• Best Couples Therapy
• Best Family Therapy
• Managing Stress
• Sleep and Dreaming
• Understanding Emotions
• Self-Improvement
• Healthy Relationships
• Student Resources
• Personality Types
• Guided Meditations
• Verywell Mind Insights
• 2024 Verywell Mind 25
• Mental Health in the Classroom
• Editorial Process
• Meet Our Review Board
• Crisis Support

The Definition of Random Assignment According to Psychology

Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

Emily is a board-certified science editor who has worked with top digital publishing brands like Voices for Biodiversity, Study.com, GoodTherapy, Vox, and Verywell.

Materio / Getty Images

Random assignment refers to the use of chance procedures in psychology experiments to ensure that each participant has the same opportunity to be assigned to any given group in a study to eliminate any potential bias in the experiment at the outset. Participants are randomly assigned to different groups, such as the treatment group versus the control group. In clinical research, randomized clinical trials are known as the gold standard for meaningful results.

Simple random assignment techniques might involve tactics such as flipping a coin, drawing names out of a hat, rolling dice, or assigning random numbers to a list of participants. It is important to note that random assignment differs from random selection .

While random selection refers to how participants are randomly chosen from a target population as representatives of that population, random assignment refers to how those chosen participants are then assigned to experimental groups.

Random Assignment In Research

To determine if changes in one variable will cause changes in another variable, psychologists must perform an experiment. Random assignment is a critical part of the experimental design that helps ensure the reliability of the study outcomes.

Researchers often begin by forming a testable hypothesis predicting that one variable of interest will have some predictable impact on another variable.

The variable that the experimenters will manipulate in the experiment is known as the independent variable , while the variable that they will then measure for different outcomes is known as the dependent variable. While there are different ways to look at relationships between variables, an experiment is the best way to get a clear idea if there is a cause-and-effect relationship between two or more variables.

Once researchers have formulated a hypothesis, conducted background research, and chosen an experimental design, it is time to find participants for their experiment. How exactly do researchers decide who will be part of an experiment? As mentioned previously, this is often accomplished through something known as random selection.

Random Selection

In order to generalize the results of an experiment to a larger group, it is important to choose a sample that is representative of the qualities found in that population. For example, if the total population is 60% female and 40% male, then the sample should reflect those same percentages.

Choosing a representative sample is often accomplished by randomly picking people from the population to be participants in a study. Random selection means that everyone in the group stands an equal chance of being chosen to minimize any bias. Once a pool of participants has been selected, it is time to assign them to groups.

By randomly assigning the participants into groups, the experimenters can be fairly sure that each group will have the same characteristics before the independent variable is applied.

Participants might be randomly assigned to the control group , which does not receive the treatment in question. The control group may receive a placebo or receive the standard treatment. Participants may also be randomly assigned to the experimental group , which receives the treatment of interest. In larger studies, there can be multiple treatment groups for comparison.

There are simple methods of random assignment, like rolling the die. However, there are more complex techniques that involve random number generators to remove any human error.

There can also be random assignment to groups with pre-established rules or parameters. For example, if you want to have an equal number of men and women in each of your study groups, you might separate your sample into two groups (by sex) before randomly assigning each of those groups into the treatment group and control group.

Random assignment is essential because it increases the likelihood that the groups are the same at the outset. With all characteristics being equal between groups, other than the application of the independent variable, any differences found between group outcomes can be more confidently attributed to the effect of the intervention.

Example of Random Assignment

Imagine that a researcher is interested in learning whether or not drinking caffeinated beverages prior to an exam will improve test performance. After randomly selecting a pool of participants, each person is randomly assigned to either the control group or the experimental group.

The participants in the control group consume a placebo drink prior to the exam that does not contain any caffeine. Those in the experimental group, on the other hand, consume a caffeinated beverage before taking the test.

Participants in both groups then take the test, and the researcher compares the results to determine if the caffeinated beverage had any impact on test performance.

A Word From Verywell

Random assignment plays an important role in the psychology research process. Not only does this process help eliminate possible sources of bias, but it also makes it easier to generalize the results of a tested sample of participants to a larger population.

Random assignment helps ensure that members of each group in the experiment are the same, which means that the groups are also likely more representative of what is present in the larger population of interest. Through the use of this technique, psychology researchers are able to study complex phenomena and contribute to our understanding of the human mind and behavior.

Lin Y, Zhu M, Su Z. The pursuit of balance: An overview of covariate-adaptive randomization techniques in clinical trials . Contemp Clin Trials. 2015;45(Pt A):21-25. doi:10.1016/j.cct.2015.07.011

Sullivan L. Random assignment versus random selection . In: The SAGE Glossary of the Social and Behavioral Sciences. SAGE Publications, Inc.; 2009. doi:10.4135/9781412972024.n2108

Alferes VR. Methods of Randomization in Experimental Design . SAGE Publications, Inc.; 2012. doi:10.4135/9781452270012

Nestor PG, Schutt RK. Research Methods in Psychology: Investigating Human Behavior. (2nd Ed.). SAGE Publications, Inc.; 2015.

By Kendra Cherry, MSEd Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

Jump to navigation

Cochrane Training

Chapter 8: assessing risk of bias in a randomized trial.

Julian PT Higgins, Jelena Savović, Matthew J Page, Roy G Elbers, Jonathan AC Sterne

Key Points:

• This chapter details version 2 of the Cochrane risk-of-bias tool for randomized trials (RoB 2), the recommended tool for use in Cochrane Reviews.
• RoB 2 is structured into a fixed set of domains of bias, focusing on different aspects of trial design, conduct and reporting.
• Each assessment using the RoB 2 tool focuses on a specific result from a randomized trial.
• Within each domain, a series of questions (‘signalling questions’) aim to elicit information about features of the trial that are relevant to risk of bias.
• A judgement about the risk of bias arising from each domain is proposed by an algorithm, based on answers to the signalling questions. Judgements can be ‘Low’, or ‘High’ risk of bias, or can express ‘Some concerns’.
• Answers to signalling questions and judgements about risk of bias should be supported by written justifications.
• The overall risk of bias for the result is the least favourable assessment across the domains of bias. Both the proposed domain-level and overall risk-of-bias judgements can be overridden by the review authors, with justification.

Cite this chapter as: Higgins JPT, Savović J, Page MJ, Elbers RG, Sterne JAC. Chapter 8: Assessing risk of bias in a randomized trial. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane, 2023. Available from www.training.cochrane.org/handbook .

8.1 Introduction

Cochrane Reviews include an assessment of the risk of bias in each included study (see Chapter 7 for a general discussion of this topic). When randomized trials are included, the recommended tool is the revised version of the Cochrane tool, known as RoB 2, described in this chapter. The RoB 2 tool provides a framework for assessing the risk of bias in a single result (an estimate of the effect of an experimental intervention compared with a comparator intervention on a particular outcome) from any type of randomized trial.

The RoB 2 tool is structured into domains through which bias might be introduced into the result. These domains were identified based on both empirical evidence and theoretical considerations. This chapter summarizes the main features of RoB 2 applied to individually randomized parallel-group trials. It describes the process of undertaking an assessment using the RoB 2 tool, summarizes the important issues for each domain of bias, and ends with a list of the key differences between RoB 2 and the earlier version of the tool. Variants of the RoB 2 tool specific to cluster-randomized trials and crossover trials are summarized in Chapter 23 .

The full guidance document for the RoB 2 tool is available at www.riskofbias.info : it summarizes the empirical evidence underlying the tool and provides detailed explanations of the concepts covered and guidance on implementation.

8.2 Overview of RoB 2

8.2.1 selecting which results to assess within the review.

Before starting an assessment of risk of bias, authors will need to select which specific results from the included trials to assess. Because trials usually contribute multiple results to a systematic review, several risk-of-bias assessments may be needed for each trial, although it is unlikely to be feasible to assess every result for every trial in the review. It is important not to select results to assess based on the likely judgements arising from the assessment. An approach that focuses on the main outcomes of the review (the results contributing to the review’s ‘Summary of findings’ table) may be the most appropriate approach (see also Chapter 7, Section 7.3.2 ).

8.2.2 Specifying the nature of the effect of interest: ‘intention-to-treat’ effects versus ‘per-protocol’ effects

Assessments for one of the RoB 2 domains, ‘Bias due to deviations from intended interventions’, differ according to whether review authors are interested in quantifying:

• the effect of assignment to the interventions at baseline, regardless of whether the interventions are received as intended (the ‘intention-to-treat effect’); or
• the effect of adhering to the interventions as specified in the trial protocol (the ‘per-protocol effect’) (Hernán and Robins 2017).

If some patients do not receive their assigned intervention or deviate from the assigned intervention after baseline, these effects will differ, and will each be of interest. For example, the estimated effect of assignment to intervention would be the most appropriate to inform a health policy question about whether to recommend an intervention in a particular health system (e.g. whether to instigate a screening programme, or whether to prescribe a new cholesterol-lowering drug), whereas the estimated effect of adhering to the intervention as specified in the trial protocol would be the most appropriate to inform a care decision by an individual patient (e.g. whether to be screened, or whether to take the new drug). Review authors should define the intervention effect in which they are interested, and apply the risk-of-bias tool appropriately to this effect.

The effect of principal interest should be specified in the review protocol: most systematic reviews are likely to address the question of assignment rather than adherence to intervention. On occasion, review authors may be interested in both effects of interest.

The effect of assignment to intervention should be estimated by an intention-to-treat (ITT) analysis that includes all randomized participants (Fergusson et al 2002). The principles of ITT analyses are (Piantadosi 2005, Menerit 2012):

• analyse participants in the intervention groups to which they were randomized, regardless of the interventions they actually received; and
• include all randomized participants in the analysis, which requires measuring all participants’ outcomes.

An ITT analysis maintains the benefit of randomization: that, on average, the intervention groups do not differ at baseline with respect to measured or unmeasured prognostic factors. Note that the term ‘intention-to-treat’ does not have a consistent definition and is used inconsistently in study reports (Hollis and Campbell 1999, Gravel et al 2007, Bell et al 2014).

Patients and other stakeholders are often interested in the effect of adhering to the intervention as described in the trial protocol (the ‘per-protocol effect’), because it relates most closely to the implications of their choice between the interventions. However, two approaches to estimation of per-protocol effects that are commonly used in randomized trials may be seriously biased. These are:

• ‘as-treated’ analyses in which participants are analysed according to the intervention they actually received, even if their randomized allocation was to a different treatment group; and
• naïve ‘per-protocol’ analyses restricted to individuals who adhered to their assigned interventions.

Each of these analyses is problematic because prognostic factors may influence whether individuals adhere to their assigned intervention. If deviations are present, it is still possible to use data from a randomised trial to derive an unbiased estimate of the effect of adhering to intervention (Hernán and Robins 2017). However, appropriate methods require strong assumptions and published applications of such methods are relatively rare to date. When authors wish to assess the risk of bias in the estimated effect of adhering to intervention, use of results based on modern statistical methods may be at lower risk of bias than results based on ‘as-treated’ or naïve per-protocol analyses.

Trial authors often estimate the effect of intervention using more than one approach. They may not explain the reasons for their choice of analysis approach, or whether their aim is to estimate the effect of assignment or adherence to intervention. We recommend that when the effect of interest is that of assignment to intervention, the trial result included in meta-analyses, and assessed for risk of bias, should be chosen according to the following order of preference:

• the result corresponding to a full ITT analysis, as defined above;
• the result corresponding to an analysis (sometimes described as a ‘modified intention-to-treat’ (mITT) analysis) that adheres to ITT principles except that participants with missing outcome data are excluded (see Section 8.4.2 ; such an analysis does not prevent bias due to missing outcome data, which is addressed in the corresponding domain of the risk-of-bias assessment);
• a result corresponding to an ‘as-treated’ or naïve ‘per-protocol’ analysis, or an analysis from which eligible trial participants were excluded.

8.2.3 Domains of bias and how they are addressed

The domains included in RoB 2 cover all types of bias that are currently understood to affect the results of randomized trials. These are:

• bias arising from the randomization process;
• bias due to deviations from intended interventions;
• bias due to missing outcome data;
• bias in measurement of the outcome; and
• bias in selection of the reported result.

Each domain is required, and no additional domains should be added. Table 8.2.a summarizes the issues addressed within each bias domain.

For each domain, the tool comprises:

• a series of ‘signalling questions’;
• a judgement about risk of bias for the domain, which is facilitated by an algorithm that maps responses to the signalling questions to a proposed judgement;
• free text boxes to justify responses to the signalling questions and risk-of-bias judgements; and
• an option to predict (and explain) the likely direction of bias.

The signalling questions aim to provide a structured approach to eliciting information relevant to an assessment of risk of bias. They seek to be reasonably factual in nature, but some may require a degree of judgement. The response options are:

• Probably yes;
• Probably no;
• No information.

To maximize their simplicity and clarity, the signalling questions are phrased such that a response of ‘Yes’ may indicate either a low or high risk of bias, depending on the most natural way to ask the question. Responses of ‘Yes’ and ‘Probably yes’ have the same implications for risk of bias, as do responses of ‘No’ and ‘Probably no’. The definitive responses (‘Yes’ and ‘No’) would typically imply that firm evidence is available in relation to the signalling question; the ‘Probably’ versions would typically imply that a judgement has been made. Although not required, if review authors wish to calculate measures of agreement (e.g. kappa statistics) for the answers to the signalling questions, we recommend treating ‘Yes’ and ‘Probably yes’ as the same response, and ‘No’ and ‘Probably no’ as the same response.

The ‘No information’ response should be used only when both (1) insufficient details are reported to permit a response of ‘Yes’, ‘Probably yes’, ‘No’ or ‘Probably no’, and (2) in the absence of these details it would be unreasonable to respond ‘Probably yes’ or ‘Probably no’ given the circumstances of the trial. For example, in the context of a large trial run by an experienced clinical trials unit for regulatory purposes, if specific information about the randomization methods is absent, it may still be reasonable to respond ‘Probably yes’ rather than ‘No information’ to the signalling question about allocation sequence concealment.

The implications of a ‘No information’ response to a signalling question differ according to the purpose of the question. If the question seeks to identify evidence of a problem, then ‘No information’ corresponds to no evidence of that problem. If the question relates to an item that is expected to be reported (such as whether any participants were lost to follow-up), then the absence of information leads to concerns about there being a problem.

A response option ‘Not applicable’ is available for signalling questions that are answered only if the response to a previous question implies that they are required.

Signalling questions should be answered independently: the answer to one question should not affect answers to other questions in the same or other domains other than through determining which subsequent questions are answered.

Once the signalling questions are answered, the next step is to reach a risk-of-bias judgement , and assign one of three levels to each domain:

• Low risk of bias;
• Some concerns; or
• High risk of bias.

The RoB 2 tool includes algorithms that map responses to signalling questions to a proposed risk-of-bias judgement for each domain (see the full documentation at www.riskofbias.info for details). The algorithms include specific mappings of each possible combination of responses to the signalling questions (including responses of ‘No information’) to judgements of low risk of bias, some concerns or high risk of bias.

Use of the word ‘judgement’ is important for the risk-of-bias assessment. The algorithms provide proposed judgements, but review authors should verify these and change them if they feel this is appropriate. In reaching final judgements, review authors should interpret ‘risk of bias’ as ‘risk of material bias’. That is, concerns should be expressed only about issues that are likely to affect the ability to draw reliable conclusions from the study.

A free text box alongside the signalling questions and judgements provides space for review authors to present supporting information for each response. In some instances, when the same information is likely to be used to answer more than one question, one text box covers more than one signalling question. Brief, direct quotations from the text of the study report should be used whenever possible. It is important that reasons are provided for any judgements that do not follow the algorithms. The tool also provides space to indicate all the sources of information about the study obtained to inform the judgements (e.g. published papers, trial registry entries, additional information from the study authors).

RoB 2 includes optional judgements of the direction of the bias for each domain and overall. For some domains, the bias is most easily thought of as being towards or away from the null. For example, high levels of switching of participants from their assigned intervention to the other intervention may have the effect of reducing the observed difference between the groups, leading to the estimated effect of adhering to intervention (see Section 8.2.2 ) being biased towards the null. For other domains, the bias is likely to favour one of the interventions being compared, implying an increase or decrease in the effect estimate depending on which intervention is favoured. Examples include manipulation of the randomization process, awareness of interventions received influencing the outcome assessment and selective reporting of results. If review authors do not have a clear rationale for judging the likely direction of the bias, they should not guess it and can leave this response blank.

Table 8.2.a Bias domains included in version 2 of the Cochrane risk-of-bias tool for randomized trials, with a summary of the issues addressed

8.2.4 Reaching an overall risk-of-bias judgement for a result

The response options for an overall risk-of-bias judgement are the same as for individual domains. Table 8.2.b shows the approach to mapping risk-of-bias judgements within domains to an overall judgement for the outcome.

Judging a result to be at a particular level of risk of bias for an individual domain implies that the result has an overall risk of bias at least this severe. Therefore, a judgement of ‘High’ risk of bias within any domain should have similar implications for the result, irrespective of which domain is being assessed. In practice this means that if the answers to the signalling questions yield a proposed judgement of ‘High’ risk of bias, the assessors should consider whether any identified problems are of sufficient concern to warrant this judgement for that result overall. If this is not the case, the appropriate action would be to override the proposed default judgement and provide justification. ‘Some concerns’ in multiple domains may lead review authors to decide on an overall judgement of ‘High’ risk of bias for that result or group of results.

Once an overall judgement has been reached for an individual trial result, this information will need to be presented in the review and reflected in the analysis and conclusions. For discussion of the presentation of risk-of-bias assessments and how they can be incorporated into analyses, see Chapter 7 . Risk-of-bias assessments also feed into one domain of the GRADE approach for assessing certainty of a body of evidence, as discussed in Chapter 14 .

Table 8.2.b Reaching an overall risk-of-bias judgement for a specific outcome

8.3 Bias arising from the randomization process

If successfully accomplished, randomization avoids the influence of either known or unknown prognostic factors (factors that predict the outcome, such as severity of illness or presence of comorbidities) on the assignment of individual participants to intervention groups. This means that, on average, each intervention group has the same prognosis before the start of intervention. If prognostic factors influence the intervention group to which participants are assigned then the estimated effect of intervention will be biased by ‘confounding’, which occurs when there are common causes of intervention group assignment and outcome. Confounding is an important potential cause of bias in intervention effect estimates from observational studies, because treatment decisions in routine care are often influenced by prognostic factors.

To randomize participants into a study, an allocation sequence that specifies how participants will be assigned to interventions is generated, based on a process that includes an element of chance. We call this allocation sequence generation . Subsequently, steps must be taken to prevent participants or trial personnel from knowing the forthcoming allocations until after recruitment has been confirmed. This process is often termed allocation sequence concealment .

Knowledge of the next assignment (e.g. if the sequence is openly posted on a bulletin board) can enable selective enrolment of participants on the basis of prognostic factors. Participants who would have been assigned to an intervention deemed to be ‘inappropriate’ may be rejected. Other participants may be directed to the ‘appropriate’ intervention, which can be accomplished by delaying their entry into the trial until the desired allocation appears. For this reason, successful allocation sequence concealment is a vital part of randomization.

Some review authors confuse allocation sequence concealment with blinding of assigned interventions during the trial. Allocation sequence concealment seeks to prevent bias in intervention assignment by preventing trial personnel and participants from knowing the allocation sequence before and until assignment. It can always be successfully implemented, regardless of the study design or clinical area (Schulz et al 1995, Jüni et al 2001). In contrast, blinding seeks to prevent bias after assignment (Jüni et al 2001, Schulz et al 2002) and cannot always be implemented. This is often the situation, for example, in trials comparing surgical with non-surgical interventions.

8.3.1 Approaches to sequence generation

Randomization with no constraints is called simple randomization or unrestricted randomization . Sometimes blocked randomization (restricted randomization) is used to ensure that the desired ratio of participants in the experimental and comparator intervention groups (e.g. 1:1) is achieved (Schulz and Grimes 2002, Schulz and Grimes 2006). This is done by ensuring that the numbers of participants assigned to each intervention group is balanced within blocks of specified size (e.g. for every 10 consecutively entered participants): the specified number of allocations to experimental and comparator intervention groups is assigned in random order within each block. If the block size is known to trial personnel and the intervention group is revealed after assignment, then the last allocation within each block can always be predicted. To avoid this problem multiple block sizes may be used, and randomly varied (random permuted blocks).

Stratified randomization , in which randomization is performed separately within subsets of participants defined by potentially important prognostic factors, such as disease severity and study centres, is also common. In practice, stratified randomization is usually performed together with blocked randomization. The purpose of combining these two procedures is to ensure that experimental and comparator groups are similar with respect to the specified prognostic factors other than intervention. If simple (rather than blocked) randomization is used in each stratum, then stratification offers no benefit, but the randomization is still valid.

Another approach that incorporates both general concepts of stratification and restricted randomization is minimization . Minimization algorithms assign the next intervention in a way that achieves the best balance between intervention groups in relation to a specified set of prognostic factors. Minimization generally includes a random element (at least for participants enrolled when the groups are balanced with respect to the prognostic factors included in the algorithm) and should be implemented along with clear strategies for allocation sequence concealment. Some methodologists are cautious about the acceptability of minimization, while others consider it to be an attractive approach (Brown et al 2005, Clark et al 2016).

8.3.2 Allocation sequence concealment and failures of randomization

If future assignments can be anticipated, leading to a failure of allocation sequence concealment, then bias can arise through selective enrolment of participants into a study, depending on their prognostic factors. Ways in which this can happen include:

• knowledge of a deterministic assignment rule, such as by alternation, date of birth or day of admission;
• knowledge of the sequence of assignments, whether randomized or not (e.g. if a sequence of random assignments is posted on the wall); and
• ability to predict assignments successfully, based on previous assignments.

The last of these can occur when blocked randomization is used and assignments are known to the recruiter after each participant is enrolled into the trial. It may then be possible to predict future assignments for some participants, particularly when blocks are of a fixed size and are not divided across multiple recruitment centres (Berger 2005).

Attempts to achieve allocation sequence concealment may be undermined in practice. For example, unsealed allocation envelopes may be opened, while translucent envelopes may be held against a bright light to reveal the contents (Schulz et al 1995, Schulz 1995, Jüni et al 2001). Personal accounts suggest that many allocation schemes have been deduced by investigators because the methods of concealment were inadequate (Schulz 1995).

The success of randomization in producing comparable groups is often examined by comparing baseline values of important prognostic factors between intervention groups. Corbett and colleagues have argued that risk-of-bias assessments should consider whether participant characteristics are balanced between intervention groups (Corbett et al 2014). The RoB 2 tool includes consideration of situations in which baseline characteristics indicate that something may have gone wrong with the randomization process. It is important that baseline imbalances that are consistent with chance are not interpreted as evidence of risk of bias. Chance imbalances are not a source of systematic bias, and the RoB 2 tool does not aim to identify imbalances in baseline variables that have arisen due to chance.

8.4 Bias due to deviations from intended interventions

This domain relates to biases that arise when there are deviations from the intended interventions. Such differences could be the administration of additional interventions that are inconsistent with the trial protocol, failure to implement the protocol interventions as intended, or non-adherence by trial participants to their assigned intervention. Biases that arise due to deviations from intended interventions are sometimes referred to as performance biases.

The intended interventions are those specified in the trial protocol. It is often intended that interventions should change or evolve in response to the health of, or events experienced by, trial participants. For example, the investigators may intend that:

• in a trial of a new drug to control symptoms of rheumatoid arthritis, participants experiencing severe toxicities should receive additional care and/or switch to an alternative drug;
• in a trial of a specified cancer drug regimen, participants whose cancer progresses should switch to a second-line intervention; or
• in a trial comparing surgical intervention with conservative management of stable angina, participants who progress to unstable angina receive surgical intervention.

Unfortunately, trial protocols may not fully specify the circumstances in which deviations from the initial intervention should occur, or distinguish changes to intervention that are consistent with the intentions of the investigators from those that should be considered as deviations from the intended intervention. For example, a cancer trial protocol may not define progression, or specify the second-line drug that should be used in patients who progress (Hernán and Scharfstein 2018). It may therefore be necessary for review authors to document changes that are and are not considered to be deviations from intended intervention. Similarly, for trials in which the comparator intervention is ‘usual care’, the protocol may not specify interventions consistent with usual care or whether they are expected to be used alongside the experimental intervention. Review authors may therefore need to document what departures from usual care will be considered as deviations from intended intervention.

8.4.1 Non-protocol interventions

Non-protocol interventions that trial participants might receive during trial follow up and that are likely to affect the outcome of interest can lead to bias in estimated intervention effects. If possible, review authors should specify potential non-protocol interventions in advance (at review protocol writing stage). Non-protocol interventions may be identified through the expert knowledge of members of the review group, via reviews of the literature, and through discussions with health professionals.

8.4.2 The role of the effect of interest

As described in Section 8.2.2 , assessments for this domain depend on the effect of interest. In RoB 2, the only deviations from the intended intervention that are addressed in relation to the effect of assignment to the intervention are those that:

• are inconsistent with the trial protocol;
• arise because of the experimental context; and
• influence the outcome.

For example, in an unblinded study participants may feel unlucky to have been assigned to the comparator group and therefore seek the experimental intervention, or other interventions that improve their prognosis. Similarly, monitoring patients randomized to a novel intervention more frequently than those randomized to standard care would increase the risk of bias, unless such monitoring was an intended part of the novel intervention. Deviations from intervention that do not arise because of the experimental context, such as a patient’s choice to stop taking their assigned medication.

To examine the effect of adhering to the interventions as specified in the trial protocol, it is important to specify what types of deviations from the intended intervention will be examined. These will be one or more of:

• how well the intervention was implemented;
• how well participants adhered to the intervention (without discontinuing or switching to another intervention);
• whether non-protocol interventions were received alongside the intended intervention and (if so) whether they were balanced across intervention groups; and
• if such deviations are present, review authors should consider whether appropriate statistical methods were used to adjust for their effects.

8.4.3 The role of blinding

Bias due to deviations from intended interventions can sometimes be reduced or avoided by implementing mechanisms that ensure the participants, carers and trial personnel (i.e. people delivering the interventions) are unaware of the interventions received. This is commonly referred to as ‘blinding’, although in some areas (including eye health) the term ‘masking’ is preferred. Blinding, if successful, should prevent knowledge of the intervention assignment from influencing contamination (application of one of the interventions in participants intended to receive the other), switches to non-protocol interventions or non-adherence by trial participants.

Trial reports often describe blinding in broad terms, such as ‘double blind’. This term makes it difficult to know who was blinded (Schulz et al 2002). Such terms are also used inconsistently (Haahr and Hróbjartsson 2006). A review of methods used for blinding highlights the variety of methods used in practice (Boutron et al 2006).

Blinding during a trial can be difficult or impossible in some contexts, for example in a trial comparing a surgical with a non-surgical intervention. Non-blinded (‘open’) trials may take other measures to avoid deviations from intended intervention, such as treating patients according to strict criteria that prevent administration of non-protocol interventions.

Lack of blinding of participants, carers or people delivering the interventions may cause bias if it leads to deviations from intended interventions. For example, low expectations of improvement among participants in the comparator group may lead them to seek and receive the experimental intervention. Such deviations from intended intervention that arise due to the experimental context can lead to bias in the estimated effects of both assignment to intervention and of adhering to intervention.

An attempt to blind participants, carers and people delivering the interventions to intervention group does not ensure successful blinding in practice. For many blinded drug trials, the side effects of the drugs allow the possible detection of the intervention being received for some participants, unless the study compares similar interventions, for example drugs with similar side effects, or uses an active placebo (Boutron et al 2006, Bello et al 2017, Jensen et al 2017).

Deducing the intervention received, for example among participants experiencing side effects that are specific to the experimental intervention, does not in itself lead to a risk of bias. As discussed, cessation of a drug intervention because of toxicity will usually not be considered a deviation from intended intervention. See the elaborations that accompany the signalling questions in the full guidance at www.riskofbias.info for further discussion of this issue.

Risk of bias in this domain may differ between outcomes, even if the same people were aware of intervention assignments during the trial. For example, knowledge of the assigned intervention may affect behaviour (such as number of clinic visits), while not having an important impact on physiology (including risk of mortality).

Blinding of outcome assessors, to avoid bias in measuring the outcome, is considered separately, in the ‘Bias in measurement of outcomes’ domain. Bias due to differential rates of dropout (withdrawal from the study) is considered in the ‘Bias due to missing outcome data’ domain.

8.4.4 Appropriate analyses

For the effect of assignment to intervention, an appropriate analysis should follow the principles of ITT (see Section 8.2.2 ). Some authors may report a ‘modified intention-to-treat’ (mITT) analysis in which participants with missing outcome data are excluded. Such an analysis may be biased because of the missing outcome data: this is addressed in the domain ‘Bias due to missing outcome data’. Note that the phrase ‘modified intention-to-treat’ is used in different ways, and may refer to inclusion of participants who received at least one dose of treatment (Abraha and Montedori 2010); our use of the term refers to missing data rather than to adherence to intervention.

Inappropriate analyses include ‘as-treated’ analyses, naïve ‘per-protocol’ analyses, and other analyses based on post-randomization exclusion of eligible trial participants on whom outcomes were measured (Hernán and Hernandez-Diaz 2012) (see also Section 8.2.2 ).

For the effect of adhering to intervention, appropriate analysis approaches are described by Hernán and Robins (Hernán and Robins 2017). Instrumental variable approaches can be used in some circumstances to estimate the effect of intervention among participants who received the assigned intervention.

8.5 Bias due to missing outcome data

Missing measurements of the outcome may lead to bias in the intervention effect estimate. Possible reasons for missing outcome data include (National Research Council 2010):

• participants withdraw from the study or cannot be located (‘loss to follow-up’ or ‘dropout’);
• participants do not attend a study visit at which outcomes should have been measured;
• participants attend a study visit but do not provide relevant data;
• data or records are lost or are unavailable for other reasons; and
• participants can no longer experience the outcome, for example because they have died.

This domain addresses risk of bias due to missing outcome data, including biases introduced by procedures used to impute, or otherwise account for, the missing outcome data.

Some participants may be excluded from an analysis for reasons other than missing outcome data. In particular, a naïve ‘per-protocol’ analysis is restricted to participants who received the intended intervention. Potential bias introduced by such analyses, or by other exclusions of eligible participants for whom outcome data are available, is addressed in the domain ‘Bias due to deviations from intended interventions’ (see Section 8.4 ).

The ITT principle of measuring outcome data on all participants (see Section 8.2.2 ) is frequently difficult or impossible to achieve in practice. Therefore, it can often only be followed by making assumptions about the missing outcome values. Even when an analysis is described as ITT, it may exclude participants with missing outcome data and be at risk of bias (such analyses may be described as ‘modified intention-to-treat’ (mITT) analyses). Therefore, assessments of risk of bias due to missing outcome data should be based on the issues addressed in the signalling questions for this domain, and not on the way that trial authors described the analysis.

8.5.1 When do missing outcome data lead to bias?

Analyses excluding individuals with missing outcome data are examples of ‘complete-case’ analyses (analyses restricted to individuals in whom there were no missing values of included variables). To understand when missing outcome data lead to bias in such analyses, we need to consider:

• the true value of the outcome in participants with missing outcome data: this is the value of the outcome that should have been measured but was not; and
• the missingness mechanism , which is the process that led to outcome data being missing.

Whether missing outcome data lead to bias in complete case analyses depends on whether the missingness mechanism is related to the true value of the outcome. Equivalently, we can consider whether the measured (non-missing) outcomes differ systematically from the missing outcomes (the true values in participants with missing outcome data). For example, consider a trial of cognitive behavioural therapy compared with usual care for depression. If participants who are more depressed are less likely to return for follow-up, then whether a measurement of depression is missing depends on its true value which implies that the measured depression outcomes will differ systematically from the true values of the missing depression outcomes.

The specific situations in which a complete case analysis suffers from bias (when there are missing data) are discussed in detail in the full guidance for the RoB 2 tool at www.riskofbias.info . In brief:

• missing outcome data will not lead to bias if missingness in the outcome is unrelated to its true value, within each intervention group;
• missing outcome data will lead to bias if missingness in the outcome depends on both the intervention group and the true value of the outcome; and
• missing outcome data will often lead to bias if missingness is related to its true value and, additionally, the effect of the experimental intervention differs from that of the comparator intervention.

8.5.2 When is the amount of missing outcome data small enough to exclude bias?

It is tempting to classify risk of bias according to the proportion of participants with missing outcome data.

Unfortunately, there is no sensible threshold for ‘small enough’ in relation to the proportion of missing outcome data.

In situations where missing outcome data lead to bias, the extent of bias will increase as the amount of missing outcome data increases. There is a tradition of regarding a proportion of less than 5% missing outcome data as ‘small’ (with corresponding implications for risk of bias), and over 20% as ‘large’. However, the potential impact of missing data on estimated intervention effects depends on the proportion of participants with missing data, the type of outcome and (for dichotomous outcome) the risk of the event. For example, consider a study of 1000 participants in the intervention group where the observed mortality is 2% for the 900 participants with outcome data (18 deaths). Even though the proportion of data missing is only 10%, if the mortality rate in the 100 missing participants is 20% (20 deaths), the overall true mortality of the intervention group would be nearly double (3.8% vs 2%) that estimated from the observed data.

8.5.3 Judging risk of bias due to missing outcome data

It is not possible to examine directly whether the chance that the outcome is missing depends on its true value: judgements of risk of bias will depend on the circumstances of the trial. Therefore, we can only be sure that there is no bias due to missing outcome data when: (1) the outcome is measured in all participants; (2) the proportion of missing outcome data is sufficiently low that any bias is too small to be of importance; or (3) sensitivity analyses (conducted by either the trial authors or the review authors) confirm that plausible values of the missing outcome data could make no important difference to the estimated intervention effect.

Indirect evidence that missing outcome data are likely to cause bias can come from examining: (1) differences between the proportion of missing outcome data in the experimental and comparator intervention groups; and (2) reasons that outcome data are missing.

If the effects of the experimental and comparator interventions on the outcome are different, and missingness in the outcome depends on its true value, then the proportion of participants with missing data is likely to differ between the intervention groups. Therefore, differing proportions of missing outcome data in the experimental and comparator intervention groups provide evidence of potential bias.

Trial reports may provide reasons why participants have missing data. For example, trials of haloperidol to treat dementia reported various reasons such as ‘lack of efficacy’, ‘adverse experience’, ‘positive response’, ‘withdrawal of consent’ and ‘patient ran away’, and ‘patient sleeping’ (Higgins et al 2008). It is likely that some of these (e.g. ‘lack of efficacy’ and ‘positive response’) are related to the true values of the missing outcome data. Therefore, these reasons increase the risk of bias if the effects of the experimental and comparator interventions differ, or if the reasons are related to intervention group (e.g. ‘adverse experience’).

In practice, our ability to assess risk of bias will be limited by the extent to which trial authors collected and reported reasons that outcome data were missing. The situation most likely to lead to bias is when reasons for missing outcome data differ between the intervention groups: for example if participants who became seriously unwell withdrew from the comparator group while participants who recovered withdrew from the experimental intervention group.

Trial authors may present statistical analyses (in addition to or instead of complete case analyses) that attempt to address the potential for bias caused by missing outcome data. Approaches include single imputation (e.g. assuming the participant had no event; last observation carried forward), multiple imputation and likelihood-based methods (see Chapter 10 , Section 10.12.2). Imputation methods are unlikely to remove or reduce the bias that occurs when missingness in the outcome depends on its true value, unless they use information additional to intervention group assignment to predict the missing values. Review authors may attempt to address missing data using sensitivity analyses, as discussed in Chapter 10, Section 10.12.3 .

8.6 Bias in measurement of the outcome

Errors in measurement of outcomes can bias intervention effect estimates. These are often referred to as measurement error (for continuous outcomes), misclassification (for dichotomous or categorical outcomes) or under-ascertainment/over-ascertainment (for events). Measurement errors may be differential or non-differential in relation to intervention assignment:

• Differential measurement errors are related to intervention assignment. Such measures are systematically different between experimental and comparator intervention groups and are less likely when outcome assessors are blinded to intervention assignment.
• Non-differential measurement errors are unrelated to intervention assignment.

This domain relates primarily to differential errors. Non-differential measurement errors are not addressed in detail.

Risk of bias in this domain depends on the following five considerations.

1. Whether the method of measuring the outcome is appropriate. Outcomes in randomized trials should be assessed using appropriate outcome measures. For example, portable blood glucose machines used by trial participants may not reliably measure below 3.1mmol, leading to an inability to detect differences in rates of severe hypoglycaemia between an insulin intervention and placebo, and under-representation of the true incidence of this adverse effect. Such a measurement would be inappropriate for this outcome.

2. Whether measurement or ascertainment of the outcome differs, or could differ, between intervention groups. The methods used to measure or ascertain outcomes should be the same across intervention groups. This is usually the case for pre-specified outcomes, but problems may arise with passive collection of outcome data, as is often the case for unexpected adverse effects. For example, in a placebo-controlled trial, severe headaches occur more frequently in participants assigned to a new drug than those assigned to placebo. These lead to more MRI scans being done in the experimental intervention group, and therefore to more diagnoses of symptomless brain tumours, even though the drug does not increase the incidence of brain tumours. Even for a pre-specified outcome measure, the nature of the intervention may lead to methods of measuring the outcome that are not comparable across intervention groups. For example, an intervention involving additional visits to a healthcare provider may lead to additional opportunities for outcome events to be identified, compared with the comparator intervention.

3. Who is the outcome assessor. The outcome assessor can be:

• the participant, when the outcome is a participant-reported outcome such as pain, quality of life, or self-completed questionnaire;
• the intervention provider, when the outcome is the result of a clinical examination, the occurrence of a clinical event or a therapeutic decision such as decision to offer a surgical intervention; or
• an observer not directly involved in the intervention provided to the participant, such as an adjudication committee, or a health professional recording outcomes for inclusion in disease registries.

4. Whether the outcome assessor is blinded to intervention assignment. Blinding of outcome assessors is often possible even when blinding of participants and personnel during the trial is not feasible. However, it is particularly difficult for participant-reported outcomes: for example, in a trial comparing surgery with medical management when the outcome is pain at 3 months. The potential for bias cannot be ignored even if the outcome assessor cannot be blinded.

5. Whether the assessment of outcome is likely to be influenced by knowledge of intervention received. For trials in which outcome assessors were not blinded, the risk of bias will depend on whether the outcome assessment involves judgement, which depends on the type of outcome. We describe most situations in Table 8.6.a .

Table 8.6.a Considerations of risk of bias in measurement of the outcome for different types of outcomes

8.7 Bias in selection of the reported result

This domain addresses bias that arises because the reported result is selected (based on its direction, magnitude or statistical significance) from among multiple intervention effect estimates that were calculated by the trial authors. Consideration of risk of bias requires distinction between:

• an outcome domain : this is a state or endpoint of interest, irrespective of how it is measured (e.g. presence or severity of depression);
• a specific outcome measurement (e.g. measurement of depression using the Hamilton rating scale 6 weeks after starting intervention); and
• an outcome analysis : this is a specific result obtained by analysing one or more outcome measurements (e.g. the difference in mean change in Hamilton rating scale scores from baseline to 6 weeks between experimental and comparator groups).

This domain does not address bias due to selective non-reporting (or incomplete reporting) of outcome domains that were measured and analysed by the trial authors (Kirkham et al 2010). For example, deaths of trial participants may be recorded by the trialists, but the reports of the trial might contain no data for deaths, or state only that the effect estimate for mortality was not statistically significant. Such bias puts the result of a synthesis at risk because results are omitted based on their direction, magnitude or statistical significance. It should therefore be addressed at the review level, as part of an integrated assessment of the risk of reporting bias (Page and Higgins 2016). For further guidance, see Chapter 7 and Chapter 13 .

Bias in selection of the reported result typically arises from a desire for findings to support vested interests or to be sufficiently noteworthy to merit publication. It can arise for both harms and benefits, although the motivations may differ. For example, in trials comparing an experimental intervention with placebo, trialists who have a preconception or vested interest in showing that the experimental intervention is beneficial and safe may be inclined to be selective in reporting efficacy estimates that are statistically significant and favourable to the experimental intervention, along with harm estimates that are not significantly different between groups. In contrast, other trialists may selectively report harm estimates that are statistically significant and unfavourable to the experimental intervention if they believe that publicizing the existence of a harm will increase their chances of publishing in a high impact journal.

This domain considers:

1. Whether the trial was analysed in accordance with a pre-specified plan that was finalized before unblinded outcome data were available for analysis. We strongly encourage review authors to attempt to retrieve the pre-specified analysis intentions for each trial (see Chapter 7, Section 7.3.1 ). Doing so allows for the identification of any outcome measures or analyses that have been omitted from, or added to, the results report, post hoc. Review authors should ideally ask the study authors to supply the study protocol and full statistical analysis plan if these are not publicly available. In addition, if outcome measures and analyses mentioned in an article, protocol or trial registration record are not reported, study authors could be asked to clarify whether those outcome measures were in fact analysed and, if so, to supply the data.

Trial protocols should describe how unexpected adverse outcomes (that potentially reflect unanticipated harms) will be collected and analysed. However, results based on spontaneously reported adverse outcomes may lead to concerns that these were selected based on the finding being noteworthy.

For some trials, the analysis intentions will not be readily available. It is still possible to assess the risk of bias in selection of the reported result. For example, outcome measures and analyses listed in the methods section of an article can be compared with those reported. Furthermore, outcome measures and analyses should be compared across different papers describing the trial.

2. Selective reporting of a particular outcome measurement (based on the results) from among estimates for multiple measurements assessed within an outcome domain. Examples include:

• reporting only one or a subset of time points at which the outcome was measured;
• use of multiple measurement instruments (e.g. pain scales) and only reporting data for the instrument with the most favourable result;
• having multiple assessors measure an outcome domain (e.g. clinician-rated and patient-rated depression scales) and only reporting data for the measure with the most favourable result; and
• reporting only the most favourable subscale (or a subset of subscales) for an instrument when measurements for other subscales were available.

3. Selective reporting of a particular analysis (based on the results) from multiple analyses estimating intervention effects for a specific outcome measurement. Examples include:

• carrying out analyses of both change scores and post-intervention scores adjusted for baseline and reporting only the more favourable analysis;
• multiple analyses of a particular outcome measurement with and without adjustment for prognostic factors (or with adjustment for different sets of prognostic factors);
• a continuously scaled outcome converted to categorical data on the basis of multiple cut-points; and
• effect estimates generated for multiple composite outcomes with full reporting of just one or a subset.

Either type of selective reporting will lead to bias if selection is based on the direction, magnitude or statistical significance of the effect estimate.

Insufficient detail in some documents may preclude full assessment of the risk of bias (e.g. trialists only state in the trial registry record that they will measure ‘pain’, without specifying the measurement scale, time point or metric that will be used). Review authors should indicate insufficient information alongside their responses to signalling questions.

8.8 Differences from the previous version of the tool

Version 2 of the tool replaces the first version, originally published in version 5 of the Handbook in 2008, and updated in 2011 (Higgins et al 2011). Research in the field has progressed, and RoB 2 reflects current understanding of how the causes of bias can influence study results, and the most appropriate ways to assess this risk.

Authors familiar with the previous version of the tool, which is used widely in Cochrane and other systematic reviews, will notice several changes:

• assessment of bias is at the level of an individual result, rather than at a study or outcome level;
• the names given to the bias domains describe more clearly the issues targeted and should reduce confusion arising from terms that are used in different ways or may be unfamiliar (such as ‘selection bias’ and ‘performance bias’) (Mansournia et al 2017);
• signalling questions have been introduced, along with algorithms to assist authors in reaching a judgement about risk of bias for each domain;
• a distinction is introduced between considering the effect of assignment to intervention and the effect of adhering to intervention, with implications for the assessment of bias due to deviations from intended interventions;
• the assessment of bias arising from the exclusion of participants from the analysis (for example, as part of a naïve ‘per-protocol’ analysis) is under the domain of bias due to deviations from the intended intervention, rather than bias due to missing outcome data;
• the concept of selective reporting of a result is distinguished from that of selective non-reporting of a result, with the latter concept removed from the tool so that it can be addressed (more appropriately) at the level of the synthesis (see Chapter 13 );
• the option to add new domains has been removed;
• an explicit process for reaching a judgement about the overall risk of bias in the result has been introduced.

Because most Cochrane Reviews published before 2019 used the first version of the tool, authors working on updating these reviews should refer to online Chapter IV for guidance on considering whether to change methodology when updating a review.

8.9 Chapter information

Authors: Julian PT Higgins, Jelena Savović, Matthew J Page, Roy G Elbers, Jonathan AC Sterne

Acknowledgements: Contributors to the development of bias domains were: Natalie Blencowe, Isabelle Boutron, Christopher Cates, Rachel Churchill, Mark Corbett, Nicky Cullum, Jonathan Emberson, Sally Hopewell, Asbjørn Hróbjartsson, Sharea Ijaz, Peter Jüni, Jamie Kirkham, Toby Lasserson, Tianjing Li, Barney Reeves, Sasha Shepperd, Ian Shrier, Lesley Stewart, Kate Tilling, Ian White, Penny Whiting. Other contributors were: Henning Keinke Andersen, Vincent Cheng, Mike Clarke, Jon Deeks, Miguel Hernán, Daniela Junqueira, Yoon Loke, Geraldine MacDonald, Alexandra McAleenan, Richard Morris, Mona Nasser, Nishith Patel, Jani Ruotsalainen, Holger Schünemann, Jayne Tierney, Sunita Vohra, Liliane Zorzela.

Funding: Development of RoB 2 was supported by the Medical Research Council (MRC) Network of Hubs for Trials Methodology Research (MR/L004933/2- N61) hosted by the MRC ConDuCT-II Hub (Collaboration and innovation for Difficult and Complex randomised controlled Trials In Invasive procedures – MR/K025643/1), by a Methods Innovation Fund grant from Cochrane and by MRC grant MR/M025209/1 . JPTH and JACS are members of the National Institute for Health Research (NIHR) Biomedical Research Centre at University Hospitals Bristol NHS Foundation Trust and the University of Bristol, and the MRC Integrative Epidemiology Unit at the University of Bristol. JPTH, JS and JACS are members of the NIHR Collaboration for Leadership in Applied Health Research and Care West (CLAHRC West) at University Hospitals Bristol NHS Foundation Trust. JPTH and JACS received funding from NIHR Senior Investigator awards NF-SI-0617-10145 and NF-SI-0611-10168, respectively. MJP received funding from an Australian National Health and Medical Research Council (NHMRC) Early Career Fellowship (1088535). The views expressed are those of the authors and not necessarily those of the National Health Service, the NIHR, the UK Department of Health and Social Care, the MRC or the Australian NHMRC.

8.10 References

Abraha I, Montedori A. Modified intention to treat reporting in randomised controlled trials: systematic review. BMJ 2010; 340 : c2697.

Bell ML, Fiero M, Horton NJ, Hsu CH. Handling missing data in RCTs; a review of the top medical journals. BMC Medical Research Methodology 2014; 14 : 118.

Bello S, Moustgaard H, Hróbjartsson A. Unreported formal assessment of unblinding occurred in 4 of 10 randomized clinical trials, unreported loss of blinding in 1 of 10 trials. Journal of Clinical Epidemiology 2017; 81 : 42-50.

Berger VW. Quantifying the magnitude of baseline covariate imbalances resulting from selection bias in randomized clinical trials. Biometrical Journal 2005; 47 : 119-127.

Boutron I, Estellat C, Guittet L, Dechartres A, Sackett DL, Hróbjartsson A, Ravaud P. Methods of blinding in reports of randomized controlled trials assessing pharmacologic treatments: a systematic review. PLoS Medicine 2006; 3 : e425.

Brown S, Thorpe H, Hawkins K, Brown J. Minimization--reducing predictability for multi-centre trials whilst retaining balance within centre. Statistics in Medicine 2005; 24 : 3715-3727.

Clark L, Fairhurst C, Torgerson DJ. Allocation concealment in randomised controlled trials: are we getting better? BMJ 2016; 355 : i5663.

Corbett MS, Higgins JPT, Woolacott NF. Assessing baseline imbalance in randomised trials: implications for the Cochrane risk of bias tool. Research Synthesis Methods 2014; 5 : 79-85.

Fergusson D, Aaron SD, Guyatt G, Hebert P. Post-randomisation exclusions: the intention to treat principle and excluding patients from analysis. BMJ 2002; 325 : 652-654.

Gravel J, Opatrny L, Shapiro S. The intention-to-treat approach in randomized controlled trials: are authors saying what they do and doing what they say? Clinical Trials (London, England) 2007; 4 : 350-356.

Haahr MT, Hróbjartsson A. Who is blinded in randomized clinical trials? A study of 200 trials and a survey of authors. Clinical Trials (London, England) 2006; 3 : 360-365.

Hernán MA, Hernandez-Diaz S. Beyond the intention-to-treat in comparative effectiveness research. Clinical Trials (London, England) 2012; 9 : 48-55.

Hernán MA, Robins JM. Per-protocol analyses of pragmatic trials. New England Journal of Medicine 2017; 377 : 1391-1398.

Hernán MA, Scharfstein D. Cautions as Regulators Move to End Exclusive Reliance on Intention to Treat. Annals of Internal Medicine 2018; 168 : 515-516.

Higgins JPT, White IR, Wood AM. Imputation methods for missing outcome data in meta-analysis of clinical trials. Clinical Trials 2008; 5 : 225-239.

Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Savović J, Schulz KF, Weeks L, Sterne JAC. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 2011; 343 : d5928.

Hollis S, Campbell F. What is meant by intention to treat analysis? Survey of published randomised controlled trials. BMJ 1999; 319 : 670-674.

Jensen JS, Bielefeldt AO, Hróbjartsson A. Active placebo control groups of pharmacological interventions were rarely used but merited serious consideration: a methodological overview. Journal of Clinical Epidemiology 2017; 87 : 35-46.

Jüni P, Altman DG, Egger M. Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ 2001; 323 : 42-46.

Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, Williamson PR. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 2010; 340 : c365.

Mansournia MA, Higgins JPT, Sterne JAC, Hernán MA. Biases in randomized trials: a conversation between trialists and epidemiologists. Epidemiology 2017; 28 : 54-59.

Menerit CL. Clinical Trials – Design, Conduct, and Analysis. Second Edition . Oxford (UK): Oxford University Press; 2012.

National Research Council. The Prevention and Treatment of Missing Data in Clinical Trials. Panel on Handling Missing Data in Clinical Trials. Committee on National Statistics, Division of Behavioral and Social Sciences and Education . Washington, DC: The National Academies Press; 2010.

Page MJ, Higgins JPT. Rethinking the assessment of risk of bias due to selective reporting: a cross-sectional study. Systematic Reviews 2016; 5 : 108.

Piantadosi S. Clinical Trials: A Methodologic perspective . 2nd ed. Hoboken (NJ): Wiley; 2005.

Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273 : 408-412.

Schulz KF. Subverting randomization in controlled trials. JAMA 1995; 274 : 1456-1458.

Schulz KF, Grimes DA. Generation of allocation sequences in randomised trials: chance, not choice. Lancet 2002; 359 : 515-519.

Schulz KF, Chalmers I, Altman DG. The landscape and lexicon of blinding in randomized trials. Annals of Internal Medicine 2002; 136 : 254-259.

Schulz KF, Grimes DA. The Lancet Handbook of Essential Concepts in Clinical Research . Edinburgh (UK): Elsevier; 2006 2006.

For permission to re-use material from the Handbook (either academic or commercial), please see here for full details.

Incorporate STEM journalism in your classroom

• Exercise type: Activity
• Topic: Science & Society
• Category: Research & Design
• Category: Diversity in STEM

How bias affects scientific research

• Download Student Worksheet

Purpose: Students will work in groups to evaluate bias in scientific research and engineering projects and to develop guidelines for minimizing potential biases.

Procedural overview: After reading the Science News for Students article “ Think you’re not biased? Think again ,” students will discuss types of bias in scientific research and how to identify it. Students will then search the Science News archive for examples of different types of bias in scientific and medical research. Students will read the National Institute of Health’s Policy on Sex as a Biological Variable and analyze how this policy works to reduce bias in scientific research on the basis of sex and gender. Based on their exploration of bias, students will discuss the benefits and limitations of research guidelines for minimizing particular types of bias and develop guidelines of their own.

Approximate class time: 2 class periods

How Bias Affects Scientific Research student guide

Computer with access to the Science News archive

Interactive meeting and screen-sharing application for virtual learning (optional)

Directions for teachers:

One of the guiding principles of scientific inquiry is objectivity. Objectivity is the idea that scientific questions, methods and results should not be affected by the personal values, interests or perspectives of researchers. However, science is a human endeavor, and experimental design and analysis of information are products of human thought processes. As a result, biases may be inadvertently introduced into scientific processes or conclusions.

In scientific circles, bias is described as any systematic deviation between the results of a study and the “truth.” Bias is sometimes described as a tendency to prefer one thing over another, or to favor one person, thing or explanation in a way that prevents objectivity or that influences the outcome of a study or the understanding of a phenomenon. Bias can be introduced in multiple points during scientific research — in the framing of the scientific question, in the experimental design, in the development or implementation of processes used to conduct the research, during collection or analysis of data, or during the reporting of conclusions.

Researchers generally recognize several different sources of bias, each of which can strongly affect the results of STEM research. Three types of bias that often occur in scientific and medical studies are researcher bias, selection bias and information bias.

Researcher bias occurs when the researcher conducting the study is in favor of a certain result. Researchers can influence outcomes through their study design choices, including who they choose to include in a study and how data are interpreted. Selection bias can be described as an experimental error that occurs when the subjects of the study do not accurately reflect the population to whom the results of the study will be applied. This commonly happens as unequal inclusion of subjects of different races, sexes or genders, ages or abilities. Information bias occurs as a result of systematic errors during the collection, recording or analysis of data.

When bias occurs, a study’s results may not accurately represent phenomena in the real world, or the results may not apply in all situations or equally for all populations. For example, if a research study does not address the full diversity of people to whom the solution will be applied, then the researchers may have missed vital information about whether and how that solution will work for a large percentage of a target population.

Bias can also affect the development of engineering solutions. For example, a new technology product tested only with teenagers or young adults who are comfortable using new technologies may have user experience issues when placed in the hands of older adults or young children.

Want to make it a virtual lesson? Post the links to the  Science News for Students article “ Think you’re not biased? Think again ,” and the National Institutes of Health information on sickle-cell disease . A link to additional resources can be provided for the students who want to know more. After students have reviewed the information at home, discuss the four questions in the setup and the sickle-cell research scenario as a class. When the students have a general understanding of bias in research, assign students to breakout rooms to look for examples of different types of bias in scientific and medical research, to discuss the Science News article “ Biomedical studies are including more female subjects (finally) ” and the National Institute of Health’s Policy on Sex as a Biological Variable and to develop bias guidelines of their own. Make sure the students have links to all articles they will need to complete their work. Bring the groups back together for an all-class discussion of the bias guidelines they write.

Assign the Science News for Students article “ Think you’re not biased? Think again ” as homework reading to introduce students to the core concepts of scientific objectivity and bias. Request that they answer the first two questions on their guide before the first class discussion on this topic. In this discussion, you will cover the idea of objective truth and introduce students to the terminology used to describe bias. Use the background information to decide what level of detail you want to give to your students.

As students discuss bias, help them understand objective and subjective data and discuss the importance of gathering both kinds of data. Explain to them how these data differ. Some phenomena — for example, body temperature, blood type and heart rate — can be objectively measured. These data tend to be quantitative. Other phenomena cannot be measured objectively and must be considered subjectively. Subjective data are based on perceptions, feelings or observations and tend to be qualitative rather than quantitative. Subjective measurements are common and essential in biomedical research, as they can help researchers understand whether a therapy changes a patient’s experience. For instance, subjective data about the amount of pain a patient feels before and after taking a medication can help scientists understand whether and how the drug works to alleviate pain. Subjective data can still be collected and analyzed in ways that attempt to minimize bias.

Try to guide student discussion to include a larger context for bias by discussing the effects of bias on understanding of an “objective truth.” How can someone’s personal views and values affect how they analyze information or interpret a situation?

To help students understand potential effects of biases, present them with the following scenario based on information from the National Institutes of Health :

Sickle-cell disease is a group of inherited disorders that cause abnormalities in red blood cells. Most of the people who have sickle-cell disease are of African descent; it also appears in populations from the Mediterranean, India and parts of Latin America. Males and females are equally likely to inherit the condition. Imagine that a therapy was developed to treat the condition, and clinical trials enlisted only male subjects of African descent. How accurately would the results of that study reflect the therapy’s effectiveness for all people who suffer from sickle-cell disease?

In the sickle-cell scenario described above, scientists will have a good idea of how the therapy works for males of African descent. But they may not be able to accurately predict how the therapy will affect female patients or patients of different races or ethnicities. Ask the students to consider how they would devise a study that addressed all the populations affected by this disease.

Before students move on, have them answer the following questions. The first two should be answered for homework and discussed in class along with the remaining questions.

1.What is bias?

In common terms, bias is a preference for or against one idea, thing or person. In scientific research, bias is a systematic deviation between observations or interpretations of data and an accurate description of a phenomenon.

2. How can biases affect the accuracy of scientific understanding of a phenomenon? How can biases affect how those results are applied?

Bias can cause the results of a scientific study to be disproportionately weighted in favor of one result or group of subjects. This can cause misunderstandings of natural processes that may make conclusions drawn from the data unreliable. Biased procedures, data collection or data interpretation can affect the conclusions scientists draw from a study and the application of those results. For example, if the subjects that participate in a study testing an engineering design do not reflect the diversity of a population, the end product may not work as well as desired for all users.

3. Describe two potential sources of bias in a scientific, medical or engineering research project. Try to give specific examples.

Researchers can intentionally or unintentionally introduce biases as a result of their attitudes toward the study or its purpose or toward the subjects or a group of subjects. Bias can also be introduced by methods of measuring, collecting or reporting data. Examples of potential sources of bias include testing a small sample of subjects, testing a group of subjects that is not diverse and looking for patterns in data to confirm ideas or opinions already held.

4. How can potential biases be identified and eliminated before, during or after a scientific study?

Students should brainstorm ways to identify sources of bias in the design of research studies. They may suggest conducting implicit bias testing or interviews before a study can be started, developing guidelines for research projects, peer review of procedures and samples/subjects before beginning a study, and peer review of data and conclusions after the study is completed and before it is published. Students may focus on the ideals of transparency and replicability of results to help reduce biases in scientific research.

Obtain and evaluate information about bias

Students will now work in small groups to select and analyze articles for different types of bias in scientific and medical research. Students will start by searching the Science News or Science News for Students archives and selecting articles that describe scientific studies or engineering design projects. If the Science News or Science News for Students articles chosen by students do not specifically cite and describe a study, students should consult the Citations at the end of the article for links to related primary research papers. Students may need to read the methods section and the conclusions of the primary research paper to better understand the project’s design and to identify potential biases. Do not assume that every scientific paper features biased research.

Student groups should evaluate the study or engineering design project outlined in the article to identify any biases in the experimental design, data collection, analysis or results. Students may need additional guidance for identifying biases. Remind them of the prior discussion about sources of bias and task them to review information about indicators of bias. Possible indicators include extreme language such as all , none or nothing ; emotional appeals rather than logical arguments; proportions of study subjects with specific characteristics such as gender, race or age; arguments that support or refute one position over another and oversimplifications or overgeneralizations. Students may also want to look for clues related to the researchers’ personal identity such as race, religion or gender. Information on political or religious points of view, sources of funding or professional affiliations may also suggest biases.

Students should also identify any deliberate attempts to reduce or eliminate bias in the project or its results. Then groups should come back together and share the results of their analysis with the class.

If students need support in searching the archives for appropriate articles, encourage groups to brainstorm search terms that may turn up related articles. Some potential search terms include bias , study , studies , experiment , engineer , new device , design , gender , sex , race , age , aging , young , old , weight , patients , survival or medical .

If you are short on time or students do not have access to the Science News or Science News for Students archive, you may want to provide articles for students to review. Some suggested articles are listed in the additional resources  below.

Once groups have selected their articles, students should answer the following questions in their groups.

1. Record the title and URL of the article and write a brief summary of the study or project.

Answers will vary, but students should accurately cite the article evaluated and summarize the study or project described in the article. Sample answer: We reviewed the Science News article “Even brain images can be biased,” which can be found at www.sciencenews.org/blog/scicurious/even-brain-images-can-be-biased. This article describes how scientific studies of human brains that involve electronic images of brains tend to include study subjects from wealthier and more highly educated households and how researchers set out to collect new data to make the database of brain images more diverse.

2. What sources of potential bias (if any) did you identify in the study or project? Describe any procedures or policies deliberately included in the study or project to eliminate biases.

The article “Even brain images can be biased” describes how scientists identified a sampling bias in studies of brain images that resulted from the way subjects were recruited. Most of these studies were conducted at universities, so many college students volunteer to participate, which resulted in the samples being skewed toward wealthier, educated, white subjects. Scientists identified a database of pediatric brain images and evaluated the diversity of the subjects in that database. They found that although the subjects in that database were more ethnically diverse than the U.S. population, the subjects were generally from wealthier households and the parents of the subjects tended to be more highly educated than average. Scientists applied statistical methods to weight the data so that study samples from the database would more accurately reflect American demographics.

3. How could any potential biases in the study or design project have affected the results or application of the results to the target population?

Scientists studying the rate of brain development in children were able to recognize the sampling bias in the brain image database. When scientists were able to apply statistical methods to ensure a better representation of socioeconomically diverse samples, they saw a different pattern in the rate of brain development in children. Scientists learned that, in general, children’s brains matured more quickly than they had previously thought. They were able to draw new conclusions about how certain factors, such as family wealth and education, affected the rate at which children’s brains developed. But the scientsits also suggested that they needed to perform additional studies with a deliberately selected group of children to ensure true diversity in the samples.

In this phase, students will review the Science News article “ Biomedical studies are including more female subjects (finally) ” and the NIH Policy on Sex as a Biological Variable , including the “ guidance document .” Students will identify how sex and gender biases may have affected the results of biomedical research before NIH instituted its policy. The students will then work with their group to recommend other policies to minimize biases in biomedical research.

To guide their development of proposed guidelines, students should answer the following questions in their groups.

1. How have sex and gender biases affected the value and application of biomedical research?

Gender and sex biases in biomedical research have diminished the accuracy and quality of research studies and reduced the applicability of results to the entire population. When girls and women are not included in research studies, the responses and therapeutic outcomes of approximately half of the target population for potential therapies remain unknown.

2. Why do you think the NIH created its policy to reduce sex and gender biases?

In the guidance document, the NIH states that “There is a growing recognition that the quality and generalizability of biomedical research depends on the consideration of key biological variables, such as sex.” The document goes on to state that many diseases and conditions affect people of both sexes, and restricting diversity of biological variables, notably sex and gender, undermines the “rigor, transparency, and generalizability of research findings.”

3. What impact has the NIH Policy on Sex as a Biological Variable had on biomedical research?

The NIH’s policy that sex is factored into research designs, analyses and reporting tries to ensure that when developing and funding biomedical research studies, researchers and institutes address potential biases in the planning stages, which helps to reduce or eliminate those biases in the final study. Including females in biomedical research studies helps to ensure that the results of biomedical research are applicable to a larger proportion of the population, expands the therapies available to girls and women and improves their health care outcomes.

4. What other policies do you think the NIH could institute to reduce biases in biomedical research? If you were to recommend one set of additional guidelines for reducing bias in biomedical research, what guidelines would you propose? Why?

Students could suggest that the NIH should have similar policies related to race, gender identity, wealth/economic status and age. Students should identify a category of bias or an underserved segment of the population that they think needs to be addressed in order to improve biomedical research and health outcomes for all people and should recommend guidelines to reduce bias related to that group. Students recommending guidelines related to race might suggest that some populations, such as African Americans, are historically underserved in terms of access to medical services and health care, and they might suggest guidelines to help reduce the disparity. Students might recommend that a certain percentage of each biomedical research project’s sample include patients of diverse racial and ethnic backgrounds.

5. What biases would your suggested policy help eliminate? How would it accomplish that goal?

Students should describe how their proposed policy would address a discrepancy in the application of biomedical research to the entire human population. Race can be considered a biological variable, like sex, and race has been connected to higher or lower incidence of certain characteristics or medical conditions, such as blood types or diabetes, which sometimes affect how the body reponds to infectious agents, drugs, procedures or other therapies. By ensuring that people from diverse racial and ethnic groups are included in biomedical research studies, scientists and medical professionals can provide better medical care to members of those populations.

Class discussion about bias guidelines

Allow each group time to present its proposed bias-reducing guideline to another group and to receive feedback. Then provide groups with time to revise their guidelines, if necessary. Act as a facilitator while students conduct the class discussion. Use this time to assess individual and group progress. Students should demonstrate an understanding of different biases that may affect patient outcomes in biomedical research studies and in practical medical settings. As part of the group discussion, have students answer the following questions.

1. Why is it important to identify and eliminate biases in research and engineering design?

The goal of most scientific research and engineering projects is to improve the quality of life and the depth of understanding of the world we live in. By eliminating biases, we can better serve the entirety of the human population and the planet .

2. Were there any guidelines that were suggested by multiple groups? How do those actions or policies help reduce bias?

Answers will depend on the guidelines developed and recommended by other groups. Groups could suggest policies related to race, gender identity, wealth/economic status and age. Each group should clearly identify how its guidelines are designed to reduce bias and improve the quality of human life.

3. Which guidelines developed by your classmates do you think would most reduce the effects of bias on research results or engineering designs? Support your selection with evidence and scientific reasoning.

Answers will depend on the guidelines developed and recommended by other groups. Students should agree that guidelines that minimize inequities and improve health care outcomes for a larger group are preferred. Guidelines addressing inequities of race and wealth/economic status are likely to expand access to improved medical care for the largest percentage of the population. People who grow up in less economically advantaged settings have specific health issues related to nutrition and their access to clean water, for instance. Ensuring that people from the lowest economic brackets are represented in biomedical research improves their access to medical care and can dramatically change the length and quality of their lives.

Possible extension

Challenge students to honestly evaluate any biases they may have. Encourage them to take an Implicit Association Test (IAT) to identify any implicit biases they may not recognize. Harvard University has an online IAT platform where students can participate in different assessments to identify preferences and biases related to sex and gender, race, religion, age, weight and other factors. You may want to challenge students to take a test before they begin the activity, and then assign students to take a test after completing the activity to see if their preferences have changed. Students can report their results to the class if they want to discuss how awareness affects the expression of bias.

Additional resources

If you want additional resources for the discussion or to provide resources for student groups, check out the links below.

Additional Science News articles:

Even brain images can be biased

Data-driven crime prediction fails to erase human bias

What we can learn from how a doctor’s race can affect Black newborns’ survival

Bias in a common health care algorithm disproportionately hurts black patients

Female rats face sex bias too

There’s no evidence that a single ‘gay gene’ exists

Positive attitudes about aging may pay off in better health

What male bias in the mammoth fossil record says about the animal’s social groups

The man flu struggle might be real, says one researcher

Scientists may work to prevent bias, but they don’t always say so

The Bias Finders

Showdown at Sex Gap

University resources:

Project Implicit (Take an Implicit Association Tests)

Catalogue of Bias

Understanding Health Research

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Research bias
• What Is Ascertainment Bias? | Definition & Examples

What Is Ascertainment Bias? | Definition & Examples

Published on 17 October 2022 by Kassiani Nikolopoulou . Revised on 18 November 2022.

Ascertainment bias occurs when some members of the target population are more likely to be included in the sample than others. Because those who are included in the sample are systematically different from the target population, the study results are biased.

Table of contents

What is ascertainment bias, ascertainment bias examples, how to prevent ascertainment bias, other types of research bias, frequently asked questions about ascertainment bias.

Ascertainment bias is a form of systematic error that occurs during data collection and analysis. It occurs when sample units are drawn in such a way that those selected are not representative of the target population.

In medical research, ascertainment bias also refers to situations where the results of a clinical trial are distorted due to knowledge about which intervention each participant is receiving.

Ascertainment bias can be introduced by:

• The person administering the intervention
• The person receiving the intervention
• The investigator assessing or analysing the outcomes
• The report writer describing the trial in detail

Ascertainment bias can influence the generalisability of your results and threaten the external validity of your findings.

There are two main sources of ascertainment bias:

• Data collection : Ascertainment bias is an inherent problem in non-probability sampling designs like convenience samples and self-selection samples. These samples are often biased, and inferences based on them are not as trustworthy as when a random sample is used.
• Lack of blinding : In experimental designs , it is important that neither the researchers nor the participants know participant group assignments. For example, if a participant knows that they are receiving a placebo , they are less likely to report benefits related to the placebo effect . As a result, the comparison between the treatment and the control group will be distorted.

As there were not enough testing kits at the time, the virus was being detected though individuals who had severe enough symptoms to go to the ER.

However, it is likely that there were many asymptomatic patients who were not tested. As testing kits became widely available, more asymptomatic patients were identified, and the death rate associated with the virus decreased.

Blinding is an important methodological feature of placebo-controlled trials to minimise research bias and maximise the validity of the research results .

The researcher then posts the participant list to a bulletin board, where anyone on the research team has access to it. Those responsible for admitting participants could see which numbers are assigned to the placebo and which ones to the active medication.

In experimental studies , ascertainment bias can be reduced by ‘blinding’ everyone involved, including those who administer the intervention, those who receive it, and those concerned with assessing and reporting the results. This is called triple blinding .

More specifically, ascertainment bias can be avoided in the following ways during the data collection phase:

• When a placebo is compared to an active treatment, the two drugs should be similar in taste, smell, and appearance. They should also be delivered using the same procedure and in the same packaging. In this way, study participants and researchers won’t realise which drug the patient is taking.
• The person arranging the randomisation (i.e., which patient takes which drug) should have no other involvement in the study. They should not reveal to anyone else involved in the study which patient is taking which drug. This also goes for researchers involved in assessing the outcomes.

This also reduces the risk of introducing other types of bias, such as demand characteristics and confirmation bias .

Keep in mind that bias can also be introduced after data collection. To reduce ascertainment bias in this phase, make sure that:

• Participants remain anonymous
• The coding of the study groups is done prior to providing the data to the researchers responsible for the analysis and reporting of the results
• The codes remain undisclosed until the process of analysis and reporting of the trial is completed

Lastly, ascertainment bias can also affect observational studies because subjects cannot be randomised. In this case, you can reduce ascertainment bias by carefully describing the inclusion and exclusion criteria used for selecting subjects or cases.

Cognitive bias

• Confirmation bias
• Baader–Meinhof phenomenon

Selection bias

• Sampling bias
• Ascertainment bias
• Attrition bias
• Self-selection bias
• Survivorship bias
• Nonresponse bias
• Undercoverage bias
• Hawthorne effect
• Observer bias
• Omitted variable bias
• Publication bias
• Pygmalion effect
• Recall bias
• Social desirability bias
• Placebo effect

Bias in research affects the validity and reliability of your findings, leading to false conclusions and a misinterpretation of the truth. This can have serious implications in areas like medical research where, for example, a new form of treatment may be evaluated.

Common types of selection bias are:

• Sampling or ascertainment bias
• Volunteer or self-selection bias
• Non-response bias

Reliability and validity are both about how well a method measures something:

• Reliability refers to the  consistency of a measure (whether the results can be reproduced under the same conditions).
• Validity   refers to the  accuracy of a measure (whether the results really do represent what they are supposed to measure).

If you are doing experimental research , you also have to consider the internal and external validity of your experiment.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Nikolopoulou, K. (2022, November 18). What Is Ascertainment Bias? | Definition & Examples. Scribbr. Retrieved 26 May 2024, from https://www.scribbr.co.uk/bias-in-research/ascertainment-bias-explained/

Is this article helpful?

Kassiani Nikolopoulou

Other students also liked, what is a double-blind study | introduction & examples, what is selection bias | definition & examples, a quick guide to experimental design | 5 steps & examples.

Cognitive Biases, Discrimination, Heuristics, Prejudice, Stereotypes, Racism, Sexism, Self-Serving Bias, Actor/Observer Bias, Change Bias

Reviewed by Psychology Today Staff

A bias is a tendency, inclination, or prejudice toward or against something or someone. Some biases are positive and helpful—like choosing to only eat foods that are considered healthy or staying away from someone who has knowingly caused harm. But biases are often based on stereotypes, rather than actual knowledge of an individual or circumstance. Whether positive or negative, such cognitive shortcuts can result in prejudgments that lead to rash decisions or discriminatory practices.

• Bias and Stereotyping
• Biases and Cognitive Errors

Bias is often characterized as stereotypes about people based on the group to which they belong and/or based on an immutable physical characteristic they possess, such as their gender , ethnicity , or sexual orientation . This type of bias can have harmful real-world outcomes. People may or may not be aware that they hold these biases.

The phenomenon of implicit bias refers to societal input that escapes conscious detection. Paying attention to helpful biases—while keeping negative, prejudicial, or accidental biases in check—requires a delicate balance between self-protection and empathy for others.

Bias is a natural inclination for or against an idea, object, group, or individual. It is often learned and is highly dependent on variables like a person’s socioeconomic status, race, ethnicity, educational background, etc. At the individual level, bias can negatively impact someone’s personal and professional relationships; at a societal level, it can lead to unfair persecution of a group, such as the Holocaust and slavery.

Starting at a young age, people will discriminate between those who are like them, their “ingroup,” and those who are not like them, “their outgroup.” On the plus side, they can gain a sense of identity and safety. However, taken to the extreme, this categorization can foster an “us-versus-them” mentality and lead to harmful prejudice .

People are naturally biased—they like certain things and dislike others, often without being fully conscious of their prejudice. Bias is acquired at a young age, often as a result of one’s upbringing. This unconscious bias becomes problematic when it causes an individual or a group to treat others poorly as a result of their gender, ethnicity, race, or other factors. 

Generally, no. Everyone has some degree of bias . It’s human nature to assign judgment based on first impressions. Also, most people have a lifetime of conditioning by schools, religious institutions, their families of origin, and the media. However, by reflecting critically on judgments and being aware of blind spots, individuals can avoid stereotyping and acting on harmful prejudice.

Telling people to “suppress prejudice” or racism often has the opposite effect. When people are trained to notice prejudiced or racist thoughts without trying to push them away, they are able to make a deliberate choice about how they behave towards others as a result. This can lead to less discrimination and reduced bias over time.

A category of biases, known as cognitive biases, are repeated patterns of thinking that can lead to inaccurate or unreasonable conclusions. Cognitive biases may help people make quicker decisions, but those decisions aren’t always accurate. Some common reasons why include flawed memory , scarce attention, natural limits on the brain’s ability to process information, emotional input, social pressures, and even aging. When assessing research—or even one's own thoughts and behaviors—it’s important to be aware of cognitive biases and attempt to counter their effects whenever possible.

When you are the actor, you are more likely to see your actions as a result of external and situational factors . Whereas, when you are observing other people, you are more likely to perceive their actions as based on internal factors (like overall disposition). This can lead to magical thinking and a lack of self-awareness.

People tend to jump at the first available piece of information and unconsciously use it to “anchor” their decision-making process , even when the information is incorrect or prejudiced. This can lead to skewed judgment and poor decision-making , especially when they don’t take the time to reason through their options.

Attribution bias occurs when someone tries to attribute reasons or motivations to the actions of others without concrete evidence to support such assumptions.

Confirmation bias refers to the brain’s tendency to search for and focus on information that supports what someone already believes, while ignoring facts that go against those beliefs, despite their relevance.

People with hindsight bias believe they should have anticipated certain outcomes , which might only be obvious now with the benefit of more knowledge and perspective. They may forget that at the time of the event, much of the information needed simply wasn’t available. They may also make unfair assumptions that other people share their experiences and expect them to come to the same conclusions.

In the Dunning-Kruger Effect , people lack the self-awareness to accurately assess their skills. They often wind up overestimating their knowledge or ability. For example, it’s not uncommon to think you’re smarter, kinder, or better at managing others than the average person.

People are more likely to attribute someone else’s actions to their personality rather than taking into account the situation they are facing. However, they rarely make this Fundamental Attribution Error when analyzing their own behavior.

The Halo Effect occurs when your positive first impression of someone colors your overall perception of them. For example, if you are struck by how beautiful someone is, you might assume they have other positive traits, like being wise or smart or brave. A negative impression, on the first hand, can lead you to assume the worst about a person, resulting in a “Reverse Halo” or “ Horns Effect .” 

People like to win, but they hate losing more. So they tend to pay more attention to negative outcomes and weigh them more heavily than positive ones when considering a decision. This negativity bias explains why we focus more on upsetting evens, and why the news seems so dire most of the time.

People tend to overestimate the likelihood of positive outcomes when they are in a good mood. Conversely, when they are feeling down, they are more likely to expect negative outcomes. In both instances, powerful emotions are driving irrational thinking .

Have you ever heard, “Don’t throw good money after bad”? That expression is based on the Sunk Cost Fallacy. Basically, when someone is aware of the time, effort, and emotional cost that’s already gone into an endeavor, they can find it difficult to change their mind or quit a longtime goal —even when it’s the healthiest choice for them.

Chemistry pulls us in, but compatibility makes us stay.

Take the advice of Nobel Prize-winning scientist Robert Lefkowitz: Commit to testing alternative narratives to explain the facts.

Gen Z protestors have been accused of ignorance and naivete, but evidence suggests they're just as informed as older generations. Their priorities may simply be different.

When we face emergencies, it's more important than ever to slow down and think clearly and carefully to resolve the situation successfully.

A riveting new book examines the history—and price—of exposing corruption in medical research.

Not being believed, especially if their stalker is a female, is one of the major barriers that victims of stalking often encounter when trying to access support and protection. 

A Personal Perspective: The U.S. Supreme Court overturned affirmative action using Asian Americans' stories. What do Asian Americans think of affirmative action?

Masculine defaults produce unattractive workplace environments for women in prestigious fields. The solution is not to train women to act like typical men.

A simple exercise can shift your attention from fearful and negative to positive. This can dramatically shift your emotional state.

Navigating the delicate balance between selflessness and self-interest. From the evolutionary roots of altruism to its modern-day manifestations.

• Find a Therapist
• Find a Treatment Center
• Find a Psychiatrist
• Find a Support Group
• Find Online Therapy
• United States
• Brooklyn, NY
• Chicago, IL
• Houston, TX
• Los Angeles, CA
• New York, NY
• Portland, OR
• San Diego, CA
• San Francisco, CA
• Seattle, WA
• Washington, DC
• Asperger's
• Bipolar Disorder
• Chronic Pain
• Eating Disorders
• Passive Aggression
• Personality
• Goal Setting
• Positive Psychology
• Stopping Smoking
• Low Sexual Desire
• Relationships
• Child Development
• Self Tests NEW
• Therapy Center
• Diagnosis Dictionary
• Types of Therapy

At any moment, someone’s aggravating behavior or our own bad luck can set us off on an emotional spiral that threatens to derail our entire day. Here’s how we can face our triggers with less reactivity so that we can get on with our lives.

• Emotional Intelligence
• Gaslighting
• Affective Forecasting
• Neuroscience

Implicit Bias (Unconscious Bias): Definition & Examples

Charlotte Ruhl

Research Assistant & Psychology Graduate

BA (Hons) Psychology, Harvard University

Charlotte Ruhl, a psychology graduate from Harvard College, boasts over six years of research experience in clinical and social psychology. During her tenure at Harvard, she contributed to the Decision Science Lab, administering numerous studies in behavioral economics and social psychology.

Learn about our Editorial Process

Saul Mcleod, PhD

Editor-in-Chief for Simply Psychology

BSc (Hons) Psychology, MRes, PhD, University of Manchester

Saul Mcleod, PhD., is a qualified psychology teacher with over 18 years of experience in further and higher education. He has been published in peer-reviewed journals, including the Journal of Clinical Psychology.

On This Page:

Implicit bias refers to the beliefs and attitudes that affect our understanding, actions and decisions in an unconscious way.

Take-home Messages

• Implicit biases are unconscious attitudes and stereotypes that can manifest in the criminal justice system, workplace, school setting, and in the healthcare system.
• Implicit bias is also known as unconscious bias or implicit social cognition.
• There are many different examples of implicit biases, ranging from categories of race, gender, and sexuality.
• These biases often arise from trying to find patterns and navigate the overwhelming stimuli in this complicated world. Culture, media, and upbringing can also contribute to the development of such biases.
• Removing these biases is a challenge, especially because we often don’t even know they exist, but research reveals potential interventions and provides hope that levels of implicit biases in the United States are decreasing.

The term implicit bias was first coined in 1995 by psychologists Mahzarin Banaji and Anthony Greenwald, who argued that social behavior is largely influenced by unconscious associations and judgments (Greenwald & Banaji, 1995).

So, what is implicit bias?

Specifically, implicit bias refers to attitudes or stereotypes that affect our understanding, actions, and decisions in an unconscious way, making them difficult to control.

Since the mid-90s, psychologists have extensively researched implicit biases, revealing that, without even knowing it, we all possess our own implicit biases.

System 1 and System 2 Thinking

Kahneman (2011) distinguishes between two types of thinking: system 1 and system 2.

• System 1 is the brain’s fast, emotional, unconscious thinking mode. This type of thinking requires little effort, but it is often error-prone. Most everyday activities (like driving, talking, cleaning, etc.) heavily use the type 1 system.
• System 2 is slow, logical, effortful, conscious thought, where reason dominates.

Implicit Bias vs. Explicit Bias

What is meant by implicit bias.

Implicit bias (unconscious bias) refers to attitudes and beliefs outside our conscious awareness and control. Implicit biases are an example of system one thinking, so we are unaware they exist (Greenwald & Krieger, 2006).

An implicit bias may counter a person’s conscious beliefs without realizing it. For example, it is possible to express explicit liking of a certain social group or approval of a certain action while simultaneously being biased against that group or action on an unconscious level.

Therefore, implicit and explicit biases might differ for the same person.

It is important to understand that implicit biases can become explicit biases. This occurs when you become consciously aware of your prejudices and beliefs. They surface in your mind, leading you to choose whether to act on or against them.

What is meant by explicit bias?

Explicit biases are biases we are aware of on a conscious level (for example, feeling threatened by another group and delivering hate speech as a result). They are an example of system 2 thinking.

It is also possible that your implicit and explicit biases differ from your neighbor, friend, or family member. Many factors can control how such biases are developed.

What Are the Implications of Unconscious Bias?

Implicit biases become evident in many different domains of society. On an interpersonal level, they can manifest in simply daily interactions.

This occurs when certain actions (or microaggressions) make others feel uncomfortable or aware of the specific prejudices you may hold against them.

Implicit Prejudice

Implicit prejudice is the automatic, unconscious attitudes or stereotypes that influence our understanding, actions, and decisions. Unlike explicit prejudice, which is consciously controlled, implicit prejudice can occur even in individuals who consciously reject prejudice and strive for impartiality.

Unconscious racial stereotypes are a major example of implicit prejudice. In other words, having an automatic preference for one race over another without being aware of this bias.

This bias can manifest in small interpersonal interactions and has broader implications in society’s legal system and many other important sectors.

Examples may include holding an implicit stereotype that associates Black individuals as violent. As a result, you may cross the street at night when you see a Black man walking in your direction without even realizing why you are crossing the street.

The action taken here is an example of a microaggression. A microaggression is a subtle, automatic, and often nonverbal that communicates hostile, derogatory, or negative prejudicial slights and insults toward any group (Pierce, 1970). Crossing the street communicates an implicit prejudice, even though you might not even be aware.

Another example of an implicit racial bias is if a Latino student is complimented by a teacher for speaking perfect English, but he is a native English speaker. Here, the teacher assumed that English would not be his first language simply because he is Latino.

Gender Stereotypes

Gender biases are another common form of implicit bias. Gender biases are the ways in which we judge men and women based on traditional feminine and masculine assigned traits.

For example, a greater assignment of fame to male than female names (Banaji & Greenwald, 1995) reveals a subconscious bias that holds men at a higher level than their female counterparts. Whether you voice the opinion that men are more famous than women is independent of this implicit gender bias.

Another common implicit gender bias regards women in STEM (science, technology, engineering, and mathematics).

In school, girls are more likely to be associated with language over math. In contrast, males are more likely to be associated with math over language (Steffens & Jelenec, 2011), revealing clear gender-related implicit biases that can ultimately go so far as to dictate future career paths.

Even if you outwardly say men and women are equally good at math, it is possible you subconsciously associate math more strongly with men without even being aware of this association.

Health Care

Healthcare is another setting where implicit biases are very present. Racial and ethnic minorities and women are subject to less accurate diagnoses, curtailed treatment options, less pain management, and worse clinical outcomes (Chapman, Kaatz, & Carnes, 2013).

Additionally, Black children are often not treated as children or given the same compassion or level of care provided for White children (Johnson et al., 2017).

It becomes evident that implicit biases infiltrate the most common sectors of society, making it all the more important to question how we can remove these biases.

LGBTQ+ Community Bias

Similar to implicit racial and gender biases, individuals may hold implicit biases against members of the LGBTQ+ community. Again, that does not necessarily mean that these opinions are voiced outwardly or even consciously recognized by the beholder, for that matter.

Rather, these biases are unconscious. A really simple example could be asking a female friend if she has a boyfriend, assuming her sexuality and that heterosexuality is the norm or default.

Instead, you could ask your friend if she is seeing someone in this specific situation. Several other forms of implicit biases fall into categories ranging from weight to ethnicity to ability that come into play in our everyday lives.

Legal System

Both law enforcement and the legal system shed light on implicit biases. An example of implicit bias functioning in law enforcement is the shooter bias – the tendency among the police to shoot Black civilians more often than White civilians, even when they are unarmed (Mekawi & Bresin, 2015).

This bias has been repeatedly tested in the laboratory setting, revealing an implicit bias against Black individuals. Blacks are also disproportionately arrested and given harsher sentences, and Black juveniles are tried as adults more often than their White peers.

Black boys are also seen as less childlike, less innocent, more culpable, more responsible for their actions, and as being more appropriate targets for police violence (Goff, 2014).

Together, these unconscious stereotypes, which are not rooted in truth, form an array of implicit biases that are extremely dangerous and utterly unjust.

Implicit biases are also visible in the workplace. One experiment that tracked the success of White and Black job applicants found that stereotypically White received 50% more callbacks than stereotypically Black names, regardless of the industry or occupation (Bertrand & Mullainathan, 2004).

This reveals another form of implicit bias: the hiring bias – Anglicized‐named applicants receiving more favorable pre‐interview impressions than other ethnic‐named applicants (Watson, Appiah, & Thornton, 2011).

We’re susceptible to bias because of these tendencies:

We tend to seek out patterns

A key reason we develop such biases is that our brains have a natural tendency to look for patterns and associations to make sense of a very complicated world.

Research shows that even before kindergarten, children already use their group membership (e.g., racial group, gender group, age group, etc.) to guide inferences about psychological and behavioral traits.

At such a young age, they have already begun seeking patterns and recognizing what distinguishes them from other groups (Baron, Dunham, Banaji, & Carey, 2014).

And not only do children recognize what sets them apart from other groups, they believe “what is similar to me is good, and what is different from me is bad” (Cameron, Alvarez, Ruble, & Fuligni, 2001).

Children aren’t just noticing how similar or dissimilar they are to others; dissimilar people are actively disliked (Aboud, 1988).

Recognizing what sets you apart from others and then forming negative opinions about those outgroups (a social group with which an individual does not identify) contributes to the development of implicit biases.

We like to take shortcuts

Another explanation is that the development of these biases is a result of the brain’s tendency to try to simplify the world.

Mental shortcuts make it faster and easier for the brain to sort through all of the overwhelming data and stimuli we are met with every second of the day. And we take mental shortcuts all the time. Rules of thumb, educated guesses, and using “common sense” are all forms of mental shortcuts.

Implicit bias is a result of taking one of these cognitive shortcuts inaccurately (Rynders, 2019). As a result, we incorrectly rely on these unconscious stereotypes to provide guidance in a very complex world.

And especially when we are under high levels of stress, we are more likely to rely on these biases than to examine all of the relevant, surrounding information (Wigboldus, Sherman, Franzese, & Knippenberg, 2004).

Social and Cultural influences

Influences from media, culture, and your individual upbringing can also contribute to the rise of implicit associations that people form about the members of social outgroups. Media has become increasingly accessible, and while that has many benefits, it can also lead to implicit biases.

The way TV portrays individuals or the language journal articles use can ingrain specific biases in our minds.

For example, they can lead us to associate Black people with criminals or females as nurses or teachers. The way you are raised can also play a huge role. One research study found that parental racial attitudes can influence children’s implicit prejudice (Sinclair, Dunn, & Lowery, 2005).

And parents are not the only figures who can influence such attitudes. Siblings, the school setting, and the culture in which you grow up can also shape your explicit beliefs and implicit biases.

Implicit Attitude Test (IAT)

What sets implicit biases apart from other forms is that they are subconscious – we don’t know if we have them.

However, researchers have developed the Implicit Association Test (IAT) tool to help reveal such biases.

The Implicit Attitude Test (IAT) is a psychological assessment to measure an individual’s unconscious biases and associations. The test measures how quickly a person associates concepts or groups (such as race or gender) with positive or negative attributes, revealing biases that may not be consciously acknowledged.

The IAT requires participants to categorize negative and positive words together with either images or words (Greenwald, McGhee, & Schwartz, 1998).

Tests are taken online and must be performed as quickly as possible, the faster you categorize certain words or faces of a category, the stronger the bias you hold about that category.

For example, the Race IAT requires participants to categorize White faces and Black faces and negative and positive words. The relative speed of association of black faces with negative words is used as an indication of the level of anti-black bias.

Professor Brian Nosek and colleagues tested more than 700,000 subjects. They found that more than 70% of White subjects more easily associated White faces with positive words and Black faces with negative words, concluding that this was evidence of implicit racial bias (Nosek, Greenwald, & Banaji, 2007).

Outside of lab testing, it is very difficult to know if we do, in fact, possess these biases. The fact that they are so hard to detect is in the very nature of this form of bias, making them very dangerous in various real-world settings.

How to Reduce Implicit Bias

Because of the harmful nature of implicit biases, it is critical to examine how we can begin to remove them.

Practicing mindfulness is one potential way, as it reduces the stress and cognitive load that otherwise leads to relying on such biases.

A 2016 study found that brief mediation decreased unconscious bias against black people and elderly people (Lueke & Gibson, 2016), providing initial insight into the usefulness of this approach and paving the way for future research on this intervention.

Adjust your perspective

Another method is perspective-taking – looking beyond your own point of view so that you can consider how someone else may think or feel about something.

Researcher Belinda Gutierrez implemented a videogame called “Fair Play,” in which players assume the role of a Black graduate student named Jamal Davis.

As Jamal, players experience subtle race bias while completing “quests” to obtain a science degree.

Gutierrez hypothesized that participants who were randomly assigned to play the game would have greater empathy for Jamal and lower implicit race bias than participants randomized to read narrative text (not perspective-taking) describing Jamal’s experience (Gutierrez, 2014), and her hypothesis was supported, illustrating the benefits of perspective taking in increasing empathy towards outgroup members.

Specific implicit bias training has been incorporated in different educational and law enforcement settings. Research has found that diversity training to overcome biases against women in STEM improved with men (Jackson, Hillard, & Schneider, 2014).

Training programs designed to target and help overcome implicit biases may also be beneficial for police officers (Plant & Peruche, 2005), but there is not enough conclusive evidence to completely support this claim. One pitfall of such training is a potential rebound effect.

Actively trying to inhibit stereotyping actually results in the bias eventually increasing more so than if it had not been initially suppressed in the first place (Macrae, Bodenhausen, Milne, & Jetten, 1994). This is very similar to the white bear problem that is discussed in many psychology curricula.

This concept refers to the psychological process whereby deliberate attempts to suppress certain thoughts make them more likely to surface (Wegner & Schneider, 2003).

Education is crucial. Understanding what implicit biases are, how they can arise how, and how to recognize them in yourself and others are all incredibly important in working towards overcoming such biases.

Learning about other cultures or outgroups and what language and behaviors may come off as offensive is critical as well. Education is a powerful tool that can extend beyond the classroom through books, media, and conversations.

On the bright side, implicit biases in the United States have been improving.

From 2007 to 2016, implicit biases have changed towards neutrality for sexual orientation, race, and skin-tone attitudes (Charlesworth & Banaji, 2019), demonstrating that it is possible to overcome these biases.

Books for further reading

As mentioned, education is extremely important. Here are a few places to get started in learning more about implicit biases:

• Biased: Uncovering the Hidden Prejudice That Shapes What We See Think and Do by Jennifer Eberhardt
• Blindspot by Anthony Greenwald and Mahzarin Banaji
• Implicit Racial Bias Across the Law by Justin Levinson and Robert Smith

Keywords and Terminology

To find materials on implicit bias and related topics, search databases and other tools using the following keywords:

Is unconscious bias the same as implicit bias?

Yes, unconscious bias is the same as implicit bias. Both terms refer to the biases we carry without awareness or conscious control, which can affect our attitudes and actions toward others.

In what ways can implicit bias impact our interactions with others?

Implicit bias can impact our interactions with others by unconsciously influencing our attitudes, behaviors, and decisions. This can lead to stereotyping, prejudice, and discrimination, even when we consciously believe in equality and fairness.

It can affect various domains of life, including workplace dynamics, healthcare provision, law enforcement, and everyday social interactions.

What are some implicit bias examples?

Some examples of implicit biases include assuming a woman is less competent than a man in a leadership role, associating certain ethnicities with criminal behavior, or believing that older people are not technologically savvy.

Other examples include perceiving individuals with disabilities as less capable or assuming that someone who is overweight is lazy or unmotivated.

Aboud, F. E. (1988). Children and prejudice . B. Blackwell.

Banaji, M. R., & Greenwald, A. G. (1995). Implicit gender stereotyping in judgments of fame. Journal of Personality and Social Psychology , 68 (2), 181.

Baron, A. S., Dunham, Y., Banaji, M., & Carey, S. (2014). Constraints on the acquisition of social category concepts. Journal of Cognition and Development , 15 (2), 238-268.

Bertrand, M., & Mullainathan, S. (2004). Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. American economic review , 94 (4), 991-1013.

Cameron, J. A., Alvarez, J. M., Ruble, D. N., & Fuligni, A. J. (2001). Children’s lay theories about ingroups and outgroups: Reconceptualizing research on prejudice. Personality and Social Psychology Review , 5 (2), 118-128.

Chapman, E. N., Kaatz, A., & Carnes, M. (2013). Physicians and implicit bias: how doctors may unwittingly perpetuate health care disparities. Journal of general internal medicine , 28 (11), 1504-1510.

Charlesworth, T. E., & Banaji, M. R. (2019). Patterns of implicit and explicit attitudes: I. Long-term change and stability from 2007 to 2016. Psychological science , 30(2), 174-192.

Goff, P. A., Jackson, M. C., Di Leone, B. A. L., Culotta, C. M., & DiTomasso, N. A. (2014). The essence of innocence: consequences of dehumanizing Black children. Journal of personality and socialpsychology,106(4), 526.

Greenwald, A. G., & Banaji, M. R. (1995). Implicit social cognition: attitudes, self-esteem, and stereotypes. Psychological review, 102(1), 4.

Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. (1998). Measuring individual differences in implicit cognition: the implicit association test. Journal of personality and social psychology , 74(6), 1464.

Greenwald, A. G., & Krieger, L. H. (2006). Implicit bias: Scientific foundations. California Law Review , 94 (4), 945-967.

Gutierrez, B., Kaatz, A., Chu, S., Ramirez, D., Samson-Samuel, C., & Carnes, M. (2014). “Fair Play”: a videogame designed to address implicit race bias through active perspective taking. Games for health journal , 3 (6), 371-378.

Jackson, S. M., Hillard, A. L., & Schneider, T. R. (2014). Using implicit bias training to improve attitudes toward women in STEM. Social Psychology of Education , 17 (3), 419-438.

Johnson, T. J., Winger, D. G., Hickey, R. W., Switzer, G. E., Miller, E., Nguyen, M. B., … & Hausmann, L. R. (2017). Comparison of physician implicit racial bias toward adults versus children. Academic pediatrics , 17 (2), 120-126.

Kahneman, D. (2011). Thinking, fast and slow . Macmillan.

Lueke, A., & Gibson, B. (2016). Brief mindfulness meditation reduces discrimination. Psychology of Consciousness: Theory, Research, and Practice , 3 (1), 34.

Macrae, C. N., Bodenhausen, G. V., Milne, A. B., & Jetten, J. (1994). Out of mind but back in sight: Stereotypes on the rebound. Journal of personality and social psychology , 67 (5), 808.

Mekawi, Y., & Bresin, K. (2015). Is the evidence from racial bias shooting task studies a smoking gun? Results from a meta-analysis. Journal of Experimental Social Psychology , 61 , 120-130.

Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2007). The Implicit Association Test at age 7: A methodological and conceptual review. Automatic processes in social thinking and behavior , 4 , 265-292.

Pierce, C. (1970). Offensive mechanisms. The black seventies , 265-282.

Plant, E. A., & Peruche, B. M. (2005). The consequences of race for police officers’ responses to criminal suspects. Psychological Science , 16 (3), 180-183.

Rynders, D. (2019). Battling Implicit Bias in the IDEA to Advocate for African American Students with Disabilities. Touro L. Rev. , 35 , 461.

Sinclair, S., Dunn, E., & Lowery, B. (2005). The relationship between parental racial attitudes and children’s implicit prejudice. Journal of Experimental Social Psychology , 41 (3), 283-289.

Steffens, M. C., & Jelenec, P. (2011). Separating implicit gender stereotypes regarding math and language: Implicit ability stereotypes are self-serving for boys and men, but not for girls and women. Sex Roles , 64(5-6), 324-335.

Watson, S., Appiah, O., & Thornton, C. G. (2011). The effect of name on pre‐interview impressions and occupational stereotypes: the case of black sales job applicants. Journal of Applied Social Psychology , 41 (10), 2405-2420.

Wegner, D. M., & Schneider, D. J. (2003). The white bear story. Psychological Inquiry , 14 (3-4), 326-329.

Wigboldus, D. H., Sherman, J. W., Franzese, H. L., & Knippenberg, A. V. (2004). Capacity and comprehension: Spontaneous stereotyping under cognitive load. Social Cognition , 22 (3), 292-309.

Further Information

Test yourself for bias.

• Project Implicit (IAT Test) From Harvard University
• Implicit Association Test From the Social Psychology Network
• Test Yourself for Hidden Bias From Teaching Tolerance
• How The Concept Of Implicit Bias Came Into Being With Dr. Mahzarin Banaji, Harvard University. Author of Blindspot: hidden biases of good people5:28 minutes; includes a transcript
• Understanding Your Racial Biases With John Dovidio, Ph.D., Yale University From the American Psychological Association11:09 minutes; includes a transcript
• Talking Implicit Bias in Policing With Jack Glaser, Goldman School of Public Policy, University of California Berkeley21:59 minutes
• Implicit Bias: A Factor in Health Communication With Dr. Winston Wong, Kaiser Permanente19:58 minutes
• Bias, Black Lives and Academic Medicine Dr. David Ansell on Your Health Radio (August 1, 2015)21:42 minutes
• Uncovering Hidden Biases Google talk with Dr. Mahzarin Banaji, Harvard University
• Impact of Implicit Bias on the Justice System 9:14 minutes
• Students Speak Up: What Bias Means to Them 2:17 minutes
• Weight Bias in Health Care From Yale University16:56 minutes
• Gender and Racial Bias In Facial Recognition Technology 4:43 minutes

Journal Articles

• An implicit bias primer Mitchell, G. (2018). An implicit bias primer. Virginia Journal of Social Policy & the Law , 25, 27–59.
• Implicit Association Test at age 7: A methodological and conceptual review Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2007). The Implicit Association Test at age 7: A methodological and conceptual review. Automatic processes in social thinking and behavior, 4 , 265-292.
• Implicit Racial/Ethnic Bias Among Health Care Professionals and Its Influence on Health Care Outcomes: A Systematic Review Hall, W. J., Chapman, M. V., Lee, K. M., Merino, Y. M., Thomas, T. W., Payne, B. K., … & Coyne-Beasley, T. (2015). Implicit racial/ethnic bias among health care professionals and its influence on health care outcomes: a systematic review. American Journal of public health, 105 (12), e60-e76.
• Reducing Racial Bias Among Health Care Providers: Lessons from Social-Cognitive Psychology Burgess, D., Van Ryn, M., Dovidio, J., & Saha, S. (2007). Reducing racial bias among health care providers: lessons from social-cognitive psychology. Journal of general internal medicine, 22 (6), 882-887.
• Integrating implicit bias into counselor education Boysen, G. A. (2010). Integrating Implicit Bias Into Counselor Education. Counselor Education & Supervision, 49 (4), 210–227.
• Cognitive Biases and Errors as Cause—and Journalistic Best Practices as Effect Christian, S. (2013). Cognitive Biases and Errors as Cause—and Journalistic Best Practices as Effect. Journal of Mass Media Ethics, 28 (3), 160–174.
• Empathy intervention to reduce implicit bias in pre-service teachers Whitford, D. K., & Emerson, A. M. (2019). Empathy Intervention to Reduce Implicit Bias in Pre-Service Teachers. Psychological Reports, 122 (2), 670–688.

Related Articles

Cognitive Psychology

Automatic Processing in Psychology: Definition & Examples

Controlled Processing in Psychology: Definition & Examples

How Ego Depletion Can Drain Your Willpower

What is the Default Mode Network?

Theories of Selective Attention in Psychology

Availability Heuristic and Decision Making

Main Navigation Menu

Edfl 3240 - media bias assignment.

• What is Media Bias?
• Fact Checker Approach
• Strong / Weak Argument
• Is it Fake News?
• Resources for Fact Checking

Education Liaison Librarian

What is media bias?

What are Biased or Pro & Con Sources? The authors of pro & con or biased articles, books, or other sources have a specific bias and are trying to persuade the reader of a specific point of view in contrast to most academic articles that typically focus on topics in an objective manner that is meant to inform the reader.

Here are some characteristics of persuasive or biased articles:

• Generally the authors do not state their agenda or tell the reader if they are for or against the topic. The reader has to determine if it is objective or persuasive.
• The authors of these opinion or pro & con articles, books, or other resource may or may not have done research on the topic. The only way to tell is if the authors lists the sources used for their research.
• If there is a list of sources used available, they are generally only those that support the agenda or argument of the author and do include those that support a different point of view.

Bias is " a   particular  tendency ,   trend,  inclination,   feeling,   or  opinion " about someone or something. When we discuss bias in media in the US, we are generally referring to conservative (also known as right) v. liberal  (also known as left)  bias, though there are many more ways to be biased and no one is truly free of bias. 

Bias differs from fake news in that fake news is specifically untrue. Biased sources don't necessarily use lies, they just don't include the whole picture, only using the facts that support their viewpoint. By using only the facts that support their cause they are giving an incomplete and therefore inaccurate picture. 

When it is appropriate to use biased sources for assignments?

Opinion or pro & con articles, books or other resources with bias are ideal to use in argumentative papers or presentations .

They can also be used for informative research assignments, but you have to be more careful so as not to produce an unintentionally biased paper or presentation that is meant to be objective.

Most importantly, you have to be able to recognize if your source is biased and if it is appropriate for your assignment. If you are not sure, ask your instructor.

Material on this page is based on:  COM Library LibGuide https://libguides.com.edu/ https://libguides.com.edu/c.php?g=649909&p=4556558 ]

• Next: Fact Checker Approach >>
• Last Updated: Feb 6, 2024 1:08 PM
• URL: https://guides.library.ucmo.edu/Media_Bias

• My View My View
• Following Following
• Saved Saved

Workday urges judge to toss bias class action over AI hiring software

• Medium Text

Sign up here.

Reporting by Daniel Wiessner in Albany, New York

Our Standards: The Thomson Reuters Trust Principles. New Tab , opens new tab

Thomson Reuters

Dan Wiessner (@danwiessner) reports on labor and employment and immigration law, including litigation and policy making. He can be reached at [email protected].

Read Next / Editor's Picks

Industry Insight Chevron

David Thomas

Mike Scarcella, David Thomas

Karen Sloan

Henry Engler

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Grad Med Educ
• v.13(5); 2021 Oct

Countering Bias in Assessment

Adelaide h. mcclintock.

Adelaide H. McClintock, MD, is Assistant Professor of Medicine, University of Washington

Tyra Fainstad

Tyra Fainstad, MD, is Visiting Associate Professor of Medicine, University of Colorado

Joshua Jauregui

Joshua Jauregui, MD, is Assistant Professor of Emergency Medicine, University of Washington

Lalena M. Yarris

Lalena M. Yarris, MD, MCR, is Professor of Emergency Medicine and Vice Chair for Faculty Development, Oregon Health & Science University, and Deputy Editor, Journal of Graduate Medical Education

The Challenge

Assessments can provide meaningful performance data from expert observers, but this information is prone to harmful bias. Such assessment bias disproportionately affects trainees who do not resemble or share identities with those doing the assessment. 1 While hundreds of cognitive biases exist, some have particularly pernicious influences when building and sustaining diversity in medicine, with subtle differences in individual performance ratings systematically perpetuating the exclusion of marginalized groups. 1

What Is Known

Bias affects our perceptions of another's knowledge, ability, professionalism, and readiness for independent practice. 2 Over repeated assessments, these biases can result in an amplification cascade, 1 a phenomenon in which small differences in assessed performance lead to larger differences in grades and selection for awards, favoring well-represented individuals and hindering underrepresented in medicine (UiM) trainees in achieving training success. 3 Eliminating harmful effects of cognitive biases requires a multilevel response. Recognizing that bias and cognitive error cannot be “trained out” of individuals, 4 systems can be put in place to create more equitable assessment. We must start by critically evaluating the tools, processes, and outcomes of our existing assessments. 3 Equitable assessments are criterion-based, encompass multiple dimensions of optimal patient care, and take place in the context of a longitudinal relationship. 5 Assessment should be reoriented from a deficit-focused lens toward a growth-focused model that uses goal setting, differentiated assessments, direct observation, and frequent feedback, both positive and critical. 5

How You Can Start TODAY

• Require anti-bias training for all supervising clinicians. Training should facilitate faculty recognition of their own biases, teach faculty to recognize biased language in their own narrative assessments, and provide skills to counter biased thinking. 2 Create an awareness of how bias influences trainee assessment and the significance of the amplification cascade.
• Establish clear criteria for competency-based assessments. Many assessment forms have already moved from normative to criterion-based assessment (eg, pass/fail) to address inequity in advancement, including the United States Medical Licensing Examination Step 1 examination and medical school clerkship grades. Assessments of physician readiness should similarly use clear criteria for “met” or “not met” to assess readiness for independent practice.
• Prioritize assessment for learning. Formative assessment should be incorporated into feedback practices that facilitate learning and progression toward safe, independent practice. Once trainees have met appropriate skill and safety metrics, assessments should shift to focus on growth, with trainee-adviser co-constructed learning goals.
• Name, reframe, and check-in. 6 Build the expectation that assessment and feedback are a bidirectional dialogue. Before providing feedback, assessors should describe their expectations and standards. Assessors should also name the presence of bias directly to trainees. Use language that acknowledges bias, reflects subjectivity of human judgement, and focuses on observed behaviors. For example, rather than “you are a good communicator,” reframe to: “ I think the patient understood your directions since they were able to repeat them back to you.” Check-in with trainees to assess whether feedback seems relevant, true to them, and action oriented.
• Reframe trainee response. Trainees may seem defensive regarding feedback. Assessors can reframe this as trainee perspectives on situational and contextual information. For example, a trainee may reply to your feedback about improving a specific communication skill with information about how they built rapport that occurred prior to coming in the room.

RIP OUT ACTION ITEMS  

• Educate faculty about adverse effects of assessment bias; provide training to recognize and address this bias.
• Reconsider the purpose of assessment. Focus on learning and professional growth for attainment of criterion-based competencies prior to independent practice.
• (Re)build assessment systems that are criterion-based, reward multiple dimensions of patient care, and promote trainee growth, all in the context of longitudinal relationships.
• Invite trainees to meaningfully participate in individual goal setting, curriculum changes, instrument design, and the processes for defining and measuring outcomes.

What You Can Do LONG TERM

• Examine your individual and program values. Use of narrowly defined, favored norms can marginalize UiM trainees. Review your work-based assessment tools and processes to discern if certain skills or traits are routinely favored and rewarded. Are growth orientation, reflective practice, or humility assessed? Where and how are these observed and measured? Does your program have multiple strategies for trainees to demonstrate competency? Seek input from trainees and patient advocacy representatives to broaden and appropriately reward multiple definitions of success and growth.
• Provide data transparency at the program level. Analyze and share program-specific assessment data, looking for differences across groups (eg, gender, race, country of origin). Scrutinize current assessment data with checkpoints at each level of data aggregation and decision-making (work-based assessment, competency decisions, entrustment, and advancement). Seek to understand and explain how work-based assessment forms, narratives, or aggregate processes might be contributing to inequities. Develop standardized processes to remedy weak points.
• Build accountability at every level of the program. Establish at least annual reviews of assessment data at the individual faculty level and incorporate them into the Annual Program Evaluation. Data reviews should be monitored by designated institutional officials and be reported to key stakeholders (eg, sponsoring institution, Accreditation Council for Graduate Medical Education). Examine narrative evaluations for gendered or other biased language using keyword searches (eg, wonderful, fabulous, good, pleasant, open, nice) or natural language processing.
• Define and track metrics of success. Once specific sources of inequity are identified, downstream events such as advancement, fellowship match, job selection, and career advancement can then be followed to ensure progress toward opportunity equity.

References and Resources for Further Reading

The weird new hiring wars

AI bots are battling it out over job searches. No matter who wins, we all lose.

When Josh Holbrook, a software engineer in Alaska, was laid off in January, he didn't expect to spend too much time looking for a new job. He certainly didn't think he'd need to relearn the job-hunt process.

A few weeks into his search, however, Holbrook found himself out of his depth. Instead of speaking with a human recruiter at a local healthcare organization, he was screened by an AI chatbot. His résumé, created nearly a decade ago in a technical format popular among academics, was incompatible with new automated recruitment platforms. He signed up for a professional service to update it in an AI-friendly format.

"The experience was completely novel," Holbrook told me. "I've never seen that before."

Over the past couple of years, job seekers have been forced to contend with incessant layoffs , a brutal recruitment market , and days of unpaid assignments. They can now add AI recruiting systems to that pile. In 2022, the Society for Human Resource Management found that about 40% of the large-scale employers it surveyed said they were already deploying AI in HR-related activities like recruitment. Rik Mistry, who consults on large-scale corporate recruitment, told Business Insider that AI is now leveraged to write job descriptions, judge an applicant's skills, power recruiting chatbots, and rate a candidate's responses. Ian Siegel, the CEO of ZipRecruiter, estimated in 2022 that nearly three-fourths of all résumés were never seen by humans.

Some job hunters have decided to fight fire with fire, turning to programs that use AI to optimize their résumés and apply to hundreds of jobs at a time. But the emerging AI-versus-AI recruitment battle is bad news for everyone. It turns hiring into a depersonalized process, it inundates hiring managers, and it reinforces weaknesses in the system it's designed to improve. And it only seems to be getting worse.

Automation in recruitment isn't new: After job sites like Monster and LinkedIn made it easy for people to apply for jobs in the early 2010s, companies adopted applicant-tracking systems to manage the deluge of online applications. Now most résumés are first seen by software designed to evaluate a person's experience and education and rank them accordingly.

The automation has helped ease the burden on overstretched recruiters — but not by much. As the stacks of digital résumés have grown amid frequent changes to remote-work policies, the recruitment hamster wheel has spun ever faster.

We know AI isn't perfect, but we have to use it as there's pressure from the higher-ups.

AI is supposed to fix this mess, saving companies time and money by outsourcing even more of the hiring process to machine-learning algorithms. In late 2019, Unilever said it had saved 100,000 hours and about $1 million in recruitment costs with the help of automated video interviews. Platforms like LinkedIn and ZipRecruiter have started using generative AI to offer candidates personalized job recommendations and let recruiters generate listings in seconds. The Google-backed recruitment-tech startup Moonhub has an AI bot that scours the internet, gathering data from places like LinkedIn and Github, to find suitable candidates. On HireVue, employers can let a bot with a set questionnaire conduct video assessments to analyze candidates' personalities . Newer startups combine these abilities in a centralized service, allowing firms to put " hiring on autopilot ."

But hiring experts Business Insider spoke with weren't convinced it's all for the best. Many fear that over time AI will make an already frustrating system worse and spawn fresh issues like ghost hires, where companies are misled into recruiting a bot masquerading as a person.

Several seasoned recruiters told me they hadn't incorporated AI into their workflow beyond auto-generating job descriptions and summarizing candidate calls. Tatiana Becker, who specializes in tech recruiting, said software that claims to match résumés with jobs lacked the nuance to do more than keyword matching — it couldn't, for instance, identify desirable candidates who came from top schools or had a history of earning strong promotions. Chatbots that Becker's boutique agency experimented with would frequently mismatch prospects and roles, ultimately pushing the prospects away.

This résumé matching "might work for applicants of more entry-level jobs," Becker told BI, "but I would worry about using it for anything else at this point."

Despite the problems, many companies are marching forward. "We know AI isn't perfect, but we have to use it as there's pressure from the higher-ups," said a recruiter in a Fortune 500 firm who spoke on the condition of anonymity to candidly discuss his company's hiring process.

Pallavi Sinha, the vice president of growth at Humanly, a startup that offers a conversational-AI hiring platform to companies like Microsoft, said that "AI in recruiting, similar to other industries, is very much at a nascent stage." But she predicted it would continue to be incorporated into hiring.

"AI isn't here to replace human interactions," she said, "but to make our jobs and lives easier — something that we'll see more and more of over time." Sinha declined to share how many applications Humanly had processed but said its chatbot service had over a "million conversations last year alone."

For candidates, though, AI has been a nightmare. Kerry McInerney, an AI researcher at the University of Cambridge, said AI increases the amount of labor for applicants, forcing them to complete puzzles and attend automated interviews just to get to the selection stage. She argued that it makes a "depersonalized process even more alienating."

Holbrook, the software engineer, wrote on LinkedIn about his frustration. "AI is too stupid to recognize transferrable skills among tech applicants," he said, adding, "I had a resume bounced because I don't have c# listed as a fluent language, even though I've dealt with c# in my jobs and have worked with plenty of languages that are 90% the same as c#."

Danielle Caldwell, a user-experience strategist in Portland, Oregon, was confused when an AI chatbot texted her to initiate the conversation about a role she had applied for. At first, she thought it was spam. After the exchange, she was left with more questions.

"There was no way to ask questions with the bot — it was a one-way experience," Caldwell said.

The University of Sussex has found that AI video interviews can be disorienting for job seekers , who behave much less naturally in the absence of a reassuring human presence. Plus, Mclnerney said, assessing a candidate's personality based on their body language and appearance is not only "reminiscent of 19th- and 20th-century racial pseudoscience but simply doesn't work." Her research has demonstrated that even things like wearing a headscarf or having a bookshelf in the background can change a personality score.

With AI you are only accelerating the brokenness of recruiting.

A cottage industry of tools has sprung up to help candidates game AI systems. One called LazyApply, for example, can apply to thousands of jobs online on your behalf for $250. "Anyone who has had to review over 50 résumés in one sitting wants to put a toothpick in their eyes," said Peter Laughter, who's been in recruitment for about three decades. With AI, he said, "you are only accelerating the brokenness of recruiting."

Bonnie Dilber, a recruiting manager at Zapier, said these services contributed to problematic behaviors on both sides of hiring. Candidates' submitting hundreds of applications leaves recruiting teams struggling to keep up and respond, which in turn pushes applicants to feel that they need to submit even more applications to stand a chance.

A more pressing issue, Dilber added, is that often these bots submit poor applications. Dilber and other recruiters told BI that some cover letters say only, "This has been submitted by [AI tool], please contact [person's email] with questions." When Aki Ito, a BI correspondent, tried using AI to apply for jobs , the system got her race wrong, made up that she spoke Spanish, and submitted an outdated cover letter.

"We had some people accidentally include ChatGPT's entire response," Hailley Griffis, the head of communications and content at Buffer, told BI. Griffis, who said she reviewed an average of 500 applications per open role, added, "There is a very distinct tone with a lot of AI content that hasn't been edited, and we saw a lot of that."

Recruiters and researchers also worry about AI's tendency to reinforce many of the recruitment industry's existing biases . Researchers from the Berkeley Haas Center for Equity, Gender and Leadership said in 2021 that of about 133 biased AI systems they analyzed, about 44% exhibited gender bias. Other recent studies have found that AI systems are prone to screening out applicants with disabilities and de-ranking résumés with names associated with Black Americans.

Unlike a human, an algorithm will never look at past hiring decisions and rectify its mistakes, said Sandra Wachter, a tech and regulation professor at the University of Oxford. It will always do as it has been taught.

Wachter, whose team developed a bias test that's been adopted by companies like IBM and Amazon, believes it's possible to use AI to make fairer decisions — but for that to happen, employers need to address systemic issues with more inclusive data and regular checks. Moonhub, for example, has a human recruiter who vets the AI's recommendations and speaks with candidates who prefer a human over a chatbot.

But until AI improves enough, humans will remain the most effective hiring systems. Becker, the tech recruiter, said humans were still critical for "getting to the heart of the candidate's decision-making process and helping them overcome apprehensions if they've gotten cold feet," as well as for "dealing with counteroffers."

David Francis, a vice president at Talent Tech Labs, a recruitment research and advisory firm, said that while "recruitment is still and will always be a very human-focused process," AI "is a tool and can be used effectively or poorly."

But when it's used poorly, both candidates and recruiters suffer. After four months of job searching, Josh Holbrook has yet to find a job.

Shubham Agarwal  is a freelance technology journalist from Ahmedabad, India, whose work has appeared in Wired, The Verge, Fast Company, and more.

About Discourse Stories

Through our Discourse journalism, Business Insider seeks to explore and illuminate the day’s most fascinating issues and ideas. Our writers provide thought-provoking perspectives, informed by analysis, reporting, and expertise. Read more Discourse stories here .

Related stories

More from Careers

Most popular

• Main content

Computer Science & Engineering

CSE 190 - TOPICS IN COMPUTER SCIENCE AND ENGINEERING (2024-2025)

Updated: May 23rd, 2024

CSE 190 - Topics in Computer Science and Engineering

CSE 190 is a topics of special interest in Computer Science and Engineering course. Topics may vary from quarter to quarter.

Prerequisites also vary per course/per instructor. Department approval required.

CSE 190 is typically offered every quarter as staffing allows.

CSE 190 may be repeated for credit a maximum of 3 times (maximum of 12 units; assuming courses taken for a different topic)

A maximum of one  CSE 190 may enrolled/waitlisted per quarter

Note: For the Fall 2023 Computer Science (CS26) curriculum, all CSE 190 courses will be labeled with a corresponding "Tag(s)" (Systems, Theory/Abstraction, and/or Applications of Computing). CSE 190 offerings before Fall 2023 are untagged but may be used as an Open CSE Elective for Computer Science majors who have changed to the FA23 CS26 curriculum.

____________________________________________________________________________

CSE 190 A00:  Human-Centered AI with Kristen Vaccaro

Prerequisite:  No course prerequisites. 

CS Curriculum Tag(s): Applications of Computing

To enroll:  Submit course clearance request via  Enrollment Authorization System (EASy)

Description:  This course provides an introduction to harnessing the power of AI so that it benefits people and communities. Topics will include: agency and initiative, fairness and bias, transparency and explainability, confidence and errors, and privacy, ethics, and trust. Students will build a number of interactive technologies powered by AI, gain practical experience with what makes them more or less usable, and learn to evaluate their impact on individuals and communities. Students will learn to think critically (but also optimistically) about what AI systems can do and how they should be integrated into society.

___________________

CSE 190 B00:  Programmers are People Too with Michael Coblenz

Prerequisite:  CSE 110 or CSE 130.  Class is unofficially co-scheduled with CSE 291 Section B00. Students will not receive credit for both classes. 

Description:  Programmers of all kinds express their ideas using programming languages. Unfortunately, languages can be hard to use correctly, resulting in lengthy development times and buggy software. How can these languages be designed to make programmers as effective as possible?

In this course, we will learn research methods for analyzing and improving the usability of programming languages. Students will apply these techniques to languages of current and historical interest, and in the process, expand their knowledge of different ways to design languages. This course is intended as preparation for conducting independent research on the usability of programming languages. The first part of the course will emphasize research methods from human-computer interaction research, with examples drawn from programming languages and development environments. The second part of the course will focus on reading research papers that describe key results from the field. The course will include homework assignments as well as a group research project.

Does lightning-rod umpire Angel Hernandez deserve his villainous reputation?

Standing at second base, Adam Rosales knew. So did the fans watching on TV and the ticket holders in the left-field bleachers. They knew what crew chief umpire Angel Hernandez should have known.

This was May 8, 2013, the game in which Hernandez became baseball’s most notorious umpire. He’d made many notable calls before this, and he’s certainly had plenty since. But this particular miss did more than any other to establish the current prevailing narrative: That he’s simply bad at his job.

Advertisement

Rosales, a light-hitting journeyman infielder for the A’s, did the improbable, crushing a game-tying solo homer with two outs in the ninth in Cleveland. The ball clearly ricocheted off a barrier above the yellow line. But it was ruled in play. The homer was obvious to anyone who watched a replay.

“All of my teammates were saying, ‘Homer, homer!’” Rosales recently recalled. “And then (manager) Bob Melvin’s reaction was pretty telling. The call was made. Obviously it was big.”

Back in 2013, there was no calling a crew in a downtown New York bunker for an official ruling. The umpires, led by Hernandez, huddled, and then exited the field to look for themselves.

After a few minutes, Hernandez emerged. He pointed toward second base. Rosales, befuddled, stayed where he was. The A’s never scored the tying run.

That moment illustrates the two viewpoints out there about Angel Hernandez, the game’s most polarizing and controversial umpire.

If you ask Hernandez, or those close to him, they’ll point to the cheap and small replay screens that rendered reviews nearly worthless. Plus, there were other umpires in the review — why didn’t they correct it? In this scenario, it was just another chapter in this misunderstood man’s career.

Then there’s the other perspective: This was obviously a home run, critical to the game, and as crew chief, he should have seen it. Hernandez, even in 2013, had a history of controversy. He had earned no benefit of the doubt. MLB itself said in a court filing years later, during Hernandez’s racial discrimination lawsuit against the league , that this incident, and Hernandez’s inability to move past it, prevented him from getting World Series assignments.

In this scenario, Hernandez only reinforced the negative perception of him held by many around the sport.

He has brought much of it on himself over his long career. Like the time he threw the hat of then-Dodgers first base coach Mariano Duncan into the stands following an argument in 2006. Or, in 2001, when he stared down ex-Chicago Bears football player Steve McMichael at a Cubs game after McMichael used the seventh-inning stretch pulpit to criticize Hernandez.

On their own, these avoidable incidents would be forgotten like the thousands of other ejections or calls that have come and gone. But together, they paint a portrait of an umpire who’s played a major role in establishing his own villainous reputation.

“I think he’s stuck in, like, a time warp, you know,” Mets broadcaster and former pitcher Ron Darling told The New York Times last year. “He’s stuck being authoritarian in a game that rarely demands it anymore.”

“Angel is bad,” said then-Rangers manager Ron Washington in 2011. “That’s all there is to it. … I’m gonna get fined for what I told Angel. And they might add to it because of what I said about Angel. But, hey, the truth is the truth.”

“I don’t understand why he’s doing these games,” former Yankees pitcher CC Sabathia said in 2018  after Hernandez had three calls overturned in one postseason game “…He’s always bad. He’s a bad umpire.”

“He needs to find another job,” four-time All-Star Ian Kinsler said in August of 2017, “he really does.”

Those who know Hernandez, and have worked with him, tend to love him. They say he’s genuine, that he checks up on his friends and sends some of them daily religious verses. That he cares about calling the game right, and wishes the vitriolic criticism would dissipate. They point to data that indicates Hernandez is not as bad as his reputation suggests.

Or at the very least, they view him in a more nuanced light than the meme that he’s become.

“Managers and umpires are alike,” said soon-to-be Hall of Fame manager Jim Leyland. “You can get out of character a bit when you have a tough situation on the field. I think we all get out of character a little bit. But I’ve always gotten along fine with Angel.”

But those who only know his calls see an ump with a large and inconsistent strike zone. Someone who makes the game about him. Someone who simply gets calls wrong at far too high a clip.

With Hernandez, the truth lies somewhere in between.

Major League Baseball declined an interview request for Hernandez, and declined to comment for this article.

“Anybody that says he’s the worst umpire in baseball doesn’t know what they’re talking about,” said Joe West, who has umpired more games than anyone ever, and has himself drawn plenty of criticism over the years.

“He does his job the right way. Does he make mistakes? Yes. But we all do. We’re not perfect. You’re judging him on every pitch. And the scrutiny on him is not fair.”

Of course, even West understands that he might not be the best person to make Hernandez’s case. “As soon as you write that Joe West says he’s a good umpire,” he said, “you’re going to get all kinds of heat.”

Hernandez’s family moved from Cuba to Florida when he was 14 months old in the early 1960s. His late father, Angel Hernandez Sr., ran a Little League in Hialeah. At 14 years old, the younger Hernandez played baseball in the Hialeah Koury League, and umpired others when his games finished. At his father’s urging, Hernandez went on to the Bill Kinnamon Umpiring School, where he was the youngest of 134 students. He finished first in the class.

When he was 20 years old, Hernandez was living out of a suitcase, making $900 a month as he traveled up and down the Florida State League. It was a grind. Each night, he’d ump another game alongside his partner, Joe Loughran.

The two drove in Loughran’s ’79 Datsun. They shared modest meals and rooms at Ramada Inns. They’d sit by the pool together.

“There was a real camaraderie there, which was a lucky thing because that’s not always the case,” Loughran said. “Maybe you have a partner who isn’t as friendly or compatible, but that was not an issue.”

Hernandez did this for more than a decade. He drove up to 30,000 miles each season. He worked winter jobs in construction and security and even had a stint as a disc jockey. He didn’t come from money and didn’t have many fallback options.

“He was very genuine through and through,” said Loughran, who soon left the profession. “(He) knew how to conduct himself, which is half of what it takes.”

But even then, Hernandez umpired with a flair that invited blowback. Rex Hudler, now a Royals broadcaster, has told a story about Hernandez ejecting nearly half his team. Players had been chirping at Hernandez, and after he issued a warning to the dugout, they put athletic tape over their mouths to mock him. Hernandez tossed the whole group.

By the time Hernandez was calling Double-A games across the Deep South, he was accustomed to vitriol from fans, including for reasons that had nothing to do with baseball.

“I remember my name over the public address, and the shots fans would take. ‘Green card.’ ‘Banana Boat,” Hernandez said in a Miami Herald article. “Those were small hick towns. North Carolina. Alabama. These were not good places to be an umpire named Angel Hernandez.”

In 1991, he finally got an MLB opportunity. This was his dream, and as Loughran said, he achieved it on “blood and guts.” But once he got to the majors, it didn’t take long for controversy to follow.

Take the July 1998 game when a red-faced Bobby Valentine, then the Mets manager, ran out of the dugout to scream at Hernandez.

Valentine claims he knew before the game even started on this July 1998 afternoon that Hernandez would have a big zone. He said he had been told that Hernandez had to catch a flight later that day — the final game before the All-Star break. Valentine’s message to his team that day was to swing, because Hernandez would look for any reason to call you out.

“He sure as heck doesn’t want to miss the plane,” Valentine recalled recently. “I’m kind of feeling for him in the dugout. You miss the flight, and have to spend a night in Atlanta. Probably miss a vacation.”

As luck would have it, the game went extras, the Mets battling the division-rival Braves in the 11th inning. Michael Tucker tagged up on a fly ball to left. The ball went to Mike Piazza at the plate, and Tucker was very clearly out.

That is, to everyone except Hernandez, who called him safe to end the game.

Valentine acknowledges now that he likes Hernandez as a person. Most of their interactions have been friendly. On that day, Valentine let Hernandez hear it.

“He didn’t mind telling you, ‘take a f—ing hike. Get out of my face,’ that type of thing,” Valentine said. “Where other guys might stand there and take it until you’re out of breath. He didn’t mind adding color to the situation.”

It’s not a coincidence that Hernandez often finds himself at the center of it all. He seems to invite it.

He infamously had a back-and-forth with Bryce Harper last season after Hernandez said the MVP went around on what was clearly a check swing.

Harper was incensed. But Hernandez appeared to respond by telling him, “You’ll see” — a cocky retort when the video would later show that it was, in fact, Hernandez who was wrong.

“It’s just bad. Just all around,” Harper later told the local media. “Angel in the middle of something again. Every year. It’s the same story. Same thing.”

In 2020, there was a similar check swing controversy. Hernandez ruled that Yankees first baseman Mike Ford went around. Then he called him out on strikes on a pitch inside.

Even in the messiest arguments with umpires, the tone and tenor rarely get personal. But Hernandez seems to engender a different type of fight.

“That’s f—ing bull—-,” then-Yankees third-base coach Phil Nevin yelled . “We all know you don’t want to be here anyway.”

Plenty of fans might understand why Nevin would feel that way. When Hernandez is behind the plate, it can seem that anything might be a strike.

Early this season, Wyatt Langford watched three consecutive J.P. France pitches land well off the outside corner — deep into the lefty batter’s box. None of the pitches to the Rangers rookie resembled a strike.

“You have got to be kidding me,” said Dave Raymond, the incensed Texas broadcaster. “What in the world?”

When it comes to egregious calls, it feels as though Hernandez is the biggest culprit. But is he the game’s worst umpire? The answer to that, statistically, is no.

According to Dylan Yep, who founded and runs Umpire Auditor since 2014, he’s ranked as the 60th to 70th best umpire, out of 85-to-90, in any given season.

“It sort of becomes a self-fulfilling prophecy, and there’s also a lot of confirmation bias,” Yep said. “When he does make a mistake, everyone is immediately tweeting about it. Everybody is tagging me. If I’m not tweeting something about it, there are a dozen other baseball accounts that will.

“Every single thing he does is scrutinized and then spread across the internet in a matter of 30 seconds.”

Even on April 12, the night he called Langford out on strikes, two other umpires had less accurate games behind the plate . Only Hernandez became a laughingstock on social media.

Yep finds Hernandez’s performances to be almost inexplicable. He’ll call a mostly normal game, Yep said, with the exception of one or two notably odd decisions — which inevitably draw attention his way.

“He consistently ends up in incredibly odd scenarios,” Yep said, “and he seems to make incorrect calls in bizarre scenarios.”

Umpire Angel Hernandez also called strikes on 7 pitches that missed by 3+ inches. This was the most in a game since 2020. https://t.co/2LvDoJaLio pic.twitter.com/jSJqkpYIbH — Umpire Auditor (@UmpireAuditor) April 13, 2024

Many of his colleagues have come to his defense over the years. After Kinsler made those aforementioned comments in 2017, umpires across the game wore white wristbands as a show of solidarity against the league’s decision not to suspend him.

Longtime umpire Ted Barrett recently posted a heartfelt defense of Hernandez on Facebook.

“He is one of the kindest men I have ever known,” Barrett wrote. “His love for his friends is immense, his love for his family is even greater. … His mistakes are magnified and sent out to the world, but his kind deeds are done in private.”

A confluence of factors have put umpires in a greater spotlight. Replay reviews overturning calls. Strike zone graphics on every broadcast. Independent umpire scorecards on social media, which Hernandez’s defenders contend are not fully accurate.

It’s all contributed, they argue, to Hernandez being the face of bad umpiring, even if it’s not deserved.

“He’s very passionate about the job, and very passionate about doing what’s right, frankly,” longtime umpire Dale Scott said. “That’s not true — the perception that he doesn’t care. That just doesn’t resonate with me.”

Still, Hernandez generally does not interact well in arguments. And his actions, including quick or haphazard ejections, don’t de-escalate those situations.

These interactions were likely a significant reason Hernandez lost the lawsuit that he filed against MLB in 2017. He alleged that he was passed over for a crew chief position and desirable postseason assignments because of his race.

The basis for the suit was a belief that MLB’s executive VP for baseball operations Joe Torre had a vendetta against Hernandez. The suit also pointed to a lack of  diversity in crew chief positions , and attorneys cited damaging deposition testimony from MLB director of umpiring Randy Marsh, who spoke about recruiting minority umpires to the profession. “The problem is, yeah, they want the job,” Marsh said, “but they want to be in the big leagues tomorrow, and they don’t want to go through all of that.”

MLB contended in its response that “Hernandez has been quick to eject managers, which inflames on-field tensions, rather than issue warnings that potentially could defuse those situations. Hernandez has also failed to communicate with other umpires on his crew, which has resulted in confusion on the field and unnecessary game delays.”

The league also said his internal evaluations consistently said he was “attempting to put himself in the spotlight.”

Essentially, MLB contended that Hernandez wasn’t equipped to handle a promotion — and because of that, and only that, he wasn’t promoted. A United States district judge agreed and granted a summary judgment in MLB’s favor.

Hernandez’s lawyer, Kevin Murphy, says the lawsuit still led to positive developments in the commissioner’s office. “That’s another thing that Angel can keep in his heart,” Murphy said. “The changes, not only with getting more opportunities for minority umpires. But he changed the commissioner’s office. Nobody’s going to give him credit for that.”

Despite its criticism of Hernandez, the league has almost no recourse to fire him, or any other umpire it feels is underperforming. The union is powerful. There are mechanisms in place, such as improvement courses, which can be required to help address deficiencies.

Even Hernandez’s performance reviews, though, paint a conflicting portrait. From 2002 to 2010, according to court documents, Hernandez received “meets standard” or “exceeds standard” ratings in all components of his performance evaluations from the league. From 2011-16, Hernandez received only one “does not meet” rating.

His 2016 year-end evaluation, however, did hint at the oddities that can accompany Hernandez’s umpiring. “You seem to miss calls in bunches,” the league advised Hernandez.

But for better or worse, the league and its fans are stuck with Hernandez for as long as he wants the job.

Hernandez isn’t on social media. By all accounts, he doesn’t pay much attention to the perpetual flow of frustration directed his way.

But, according to his lawyer, there are people close to Hernandez who feel the impact.

“What hurts him the most,” Murphy said, “is the pain that his two daughters and his wife go through when they know it’s so unbelievably undeserved.”

“I think it bothers him that his family has to put up with it,” West said. “He’s such a strong-character person; he doesn’t let the media affect him.”

It’s not only other umpires who have defended him. Take Homer Bailey, the former Reds pitcher who threw a no-hitter in 2012. Hernandez, the third-base umpire that night, asked for some signed baseballs following Bailey’s achievement. Bailey agreed, without issue. Hernandez would receive his one “does not meet” rating on his year-end evaluation because of it. But Bailey said the entire thing was innocuous.

“He didn’t ask for more than any of the other umpires,” Bailey said. “…Maybe there are some things he could do on his end to kind of tamp it down. But there’s also some things that get blown out of proportion.”

Hernandez is a public figure in a major professional sport, and criticism is baked into officiating. But how much of it is justified?

Leyland will turn 80 years old this year — just a few months after his formal Hall of Fame induction. His interactions with Hernandez are long in the past.

With that age, and those 22 years as a skipper, has come some perspective.

“A manager, half the games, he has the home crowd behind him. Normally, you’ve got a home base,” Leyland said. “The umpire doesn’t have a home base. He’s a stranger. He’s on the road every night. He doesn’t have a hometown.

“We all know they miss calls. But we also all know that when you look at all the calls that are made in a baseball season by the umpires, they’re goddamn good. They’re really good at what they do.”

Leyland has found what so few others have been able to: A nuanced perspective on Hernandez.

For almost everyone else, that seems to be impossible.

The Athletic ’s Chad Jennings contributed to this story

(Top image: Sean Reilly / The Athletic ; Photos: Jamie Squire / Getty Images; Jason O. Watson / Getty Images; Tom Szczerbowski / Getty Images)

Get all-access to exclusive stories.

Subscribe to The Athletic for in-depth coverage of your favorite players, teams, leagues and clubs. Try a week on us.