what is the alternative hypothesis for the test of significance

Hypothesis Testing (cont...)

Hypothesis testing, the null and alternative hypothesis.

In order to undertake hypothesis testing you need to express your research hypothesis as a null and alternative hypothesis. The null hypothesis and alternative hypothesis are statements regarding the differences or effects that occur in the population. You will use your sample to test which statement (i.e., the null hypothesis or alternative hypothesis) is most likely (although technically, you test the evidence against the null hypothesis). So, with respect to our teaching example, the null and alternative hypothesis will reflect statements about all statistics students on graduate management courses.

The null hypothesis is essentially the "devil's advocate" position. That is, it assumes that whatever you are trying to prove did not happen ( hint: it usually states that something equals zero). For example, the two different teaching methods did not result in different exam performances (i.e., zero difference). Another example might be that there is no relationship between anxiety and athletic performance (i.e., the slope is zero). The alternative hypothesis states the opposite and is usually the hypothesis you are trying to prove (e.g., the two different teaching methods did result in different exam performances). Initially, you can state these hypotheses in more general terms (e.g., using terms like "effect", "relationship", etc.), as shown below for the teaching methods example:

Depending on how you want to "summarize" the exam performances will determine how you might want to write a more specific null and alternative hypothesis. For example, you could compare the mean exam performance of each group (i.e., the "seminar" group and the "lectures-only" group). This is what we will demonstrate here, but other options include comparing the distributions , medians , amongst other things. As such, we can state:

Now that you have identified the null and alternative hypotheses, you need to find evidence and develop a strategy for declaring your "support" for either the null or alternative hypothesis. We can do this using some statistical theory and some arbitrary cut-off points. Both these issues are dealt with next.

Significance levels

The level of statistical significance is often expressed as the so-called p -value . Depending on the statistical test you have chosen, you will calculate a probability (i.e., the p -value) of observing your sample results (or more extreme) given that the null hypothesis is true . Another way of phrasing this is to consider the probability that a difference in a mean score (or other statistic) could have arisen based on the assumption that there really is no difference. Let us consider this statement with respect to our example where we are interested in the difference in mean exam performance between two different teaching methods. If there really is no difference between the two teaching methods in the population (i.e., given that the null hypothesis is true), how likely would it be to see a difference in the mean exam performance between the two teaching methods as large as (or larger than) that which has been observed in your sample?

So, you might get a p -value such as 0.03 (i.e., p = .03). This means that there is a 3% chance of finding a difference as large as (or larger than) the one in your study given that the null hypothesis is true. However, you want to know whether this is "statistically significant". Typically, if there was a 5% or less chance (5 times in 100 or less) that the difference in the mean exam performance between the two teaching methods (or whatever statistic you are using) is as different as observed given the null hypothesis is true, you would reject the null hypothesis and accept the alternative hypothesis. Alternately, if the chance was greater than 5% (5 times in 100 or more), you would fail to reject the null hypothesis and would not accept the alternative hypothesis. As such, in this example where p = .03, we would reject the null hypothesis and accept the alternative hypothesis. We reject it because at a significance level of 0.03 (i.e., less than a 5% chance), the result we obtained could happen too frequently for us to be confident that it was the two teaching methods that had an effect on exam performance.

Whilst there is relatively little justification why a significance level of 0.05 is used rather than 0.01 or 0.10, for example, it is widely used in academic research. However, if you want to be particularly confident in your results, you can set a more stringent level of 0.01 (a 1% chance or less; 1 in 100 chance or less).

One- and two-tailed predictions

When considering whether we reject the null hypothesis and accept the alternative hypothesis, we need to consider the direction of the alternative hypothesis statement. For example, the alternative hypothesis that was stated earlier is:

The alternative hypothesis tells us two things. First, what predictions did we make about the effect of the independent variable(s) on the dependent variable(s)? Second, what was the predicted direction of this effect? Let's use our example to highlight these two points.

Sarah predicted that her teaching method (independent variable: teaching method), whereby she not only required her students to attend lectures, but also seminars, would have a positive effect (that is, increased) students' performance (dependent variable: exam marks). If an alternative hypothesis has a direction (and this is how you want to test it), the hypothesis is one-tailed. That is, it predicts direction of the effect. If the alternative hypothesis has stated that the effect was expected to be negative, this is also a one-tailed hypothesis.

Alternatively, a two-tailed prediction means that we do not make a choice over the direction that the effect of the experiment takes. Rather, it simply implies that the effect could be negative or positive. If Sarah had made a two-tailed prediction, the alternative hypothesis might have been:

In other words, we simply take out the word "positive", which implies the direction of our effect. In our example, making a two-tailed prediction may seem strange. After all, it would be logical to expect that "extra" tuition (going to seminar classes as well as lectures) would either have a positive effect on students' performance or no effect at all, but certainly not a negative effect. However, this is just our opinion (and hope) and certainly does not mean that we will get the effect we expect. Generally speaking, making a one-tail prediction (i.e., and testing for it this way) is frowned upon as it usually reflects the hope of a researcher rather than any certainty that it will happen. Notable exceptions to this rule are when there is only one possible way in which a change could occur. This can happen, for example, when biological activity/presence in measured. That is, a protein might be "dormant" and the stimulus you are using can only possibly "wake it up" (i.e., it cannot possibly reduce the activity of a "dormant" protein). In addition, for some statistical tests, one-tailed tests are not possible.

Rejecting or failing to reject the null hypothesis

Let's return finally to the question of whether we reject or fail to reject the null hypothesis.

If our statistical analysis shows that the significance level is below the cut-off value we have set (e.g., either 0.05 or 0.01), we reject the null hypothesis and accept the alternative hypothesis. Alternatively, if the significance level is above the cut-off value, we fail to reject the null hypothesis and cannot accept the alternative hypothesis. You should note that you cannot accept the null hypothesis, but only find evidence against it.

9.1 Null and Alternative Hypotheses

The actual test begins by considering two hypotheses . They are called the null hypothesis and the alternative hypothesis . These hypotheses contain opposing viewpoints.

H 0 , the — null hypothesis: a statement of no difference between sample means or proportions or no difference between a sample mean or proportion and a population mean or proportion. In other words, the difference equals 0.

H a —, the alternative hypothesis: a claim about the population that is contradictory to H 0 and what we conclude when we reject H 0 .

Since the null and alternative hypotheses are contradictory, you must examine evidence to decide if you have enough evidence to reject the null hypothesis or not. The evidence is in the form of sample data.

After you have determined which hypothesis the sample supports, you make a decision. There are two options for a decision. They are reject H 0 if the sample information favors the alternative hypothesis or do not reject H 0 or decline to reject H 0 if the sample information is insufficient to reject the null hypothesis.

Mathematical Symbols Used in H 0 and H a :

H 0 always has a symbol with an equal in it. H a never has a symbol with an equal in it. The choice of symbol depends on the wording of the hypothesis test. However, be aware that many researchers use = in the null hypothesis, even with > or < as the symbol in the alternative hypothesis. This practice is acceptable because we only make the decision to reject or not reject the null hypothesis.

Example 9.1

H 0 : No more than 30 percent of the registered voters in Santa Clara County voted in the primary election. p ≤ 30 H a : More than 30 percent of the registered voters in Santa Clara County voted in the primary election. p > 30

A medical trial is conducted to test whether or not a new medicine reduces cholesterol by 25 percent. State the null and alternative hypotheses.

Example 9.2

We want to test whether the mean GPA of students in American colleges is different from 2.0 (out of 4.0). The null and alternative hypotheses are the following: H 0 : μ = 2.0 H a : μ ≠ 2.0

We want to test whether the mean height of eighth graders is 66 inches. State the null and alternative hypotheses. Fill in the correct symbol (=, ≠, ≥, <, ≤, >) for the null and alternative hypotheses.

• H 0 : μ __ 66
• H a : μ __ 66

Example 9.3

We want to test if college students take fewer than five years to graduate from college, on the average. The null and alternative hypotheses are the following: H 0 : μ ≥ 5 H a : μ < 5

We want to test if it takes fewer than 45 minutes to teach a lesson plan. State the null and alternative hypotheses. Fill in the correct symbol ( =, ≠, ≥, <, ≤, >) for the null and alternative hypotheses.

• H 0 : μ __ 45
• H a : μ __ 45

Example 9.4

An article on school standards stated that about half of all students in France, Germany, and Israel take advanced placement exams and a third of the students pass. The same article stated that 6.6 percent of U.S. students take advanced placement exams and 4.4 percent pass. Test if the percentage of U.S. students who take advanced placement exams is more than 6.6 percent. State the null and alternative hypotheses. H 0 : p ≤ 0.066 H a : p > 0.066

On a state driver’s test, about 40 percent pass the test on the first try. We want to test if more than 40 percent pass on the first try. Fill in the correct symbol (=, ≠, ≥, <, ≤, >) for the null and alternative hypotheses.

• H 0 : p __ 0.40
• H a : p __ 0.40

Collaborative Exercise

Bring to class a newspaper, some news magazines, and some internet articles. In groups, find articles from which your group can write null and alternative hypotheses. Discuss your hypotheses with the rest of the class.

As an Amazon Associate we earn from qualifying purchases.

This book may not be used in the training of large language models or otherwise be ingested into large language models or generative AI offerings without OpenStax's permission.

Want to cite, share, or modify this book? This book uses the Creative Commons Attribution License and you must attribute Texas Education Agency (TEA). The original material is available at: https://www.texasgateway.org/book/tea-statistics . Changes were made to the original material, including updates to art, structure, and other content updates.

Access for free at https://openstax.org/books/statistics/pages/1-introduction

• Authors: Barbara Illowsky, Susan Dean
• Publisher/website: OpenStax
• Book title: Statistics
• Publication date: Mar 27, 2020
• Location: Houston, Texas
• Book URL: https://openstax.org/books/statistics/pages/1-introduction
• Section URL: https://openstax.org/books/statistics/pages/9-1-null-and-alternative-hypotheses

© Jan 23, 2024 Texas Education Agency (TEA). The OpenStax name, OpenStax logo, OpenStax book covers, OpenStax CNX name, and OpenStax CNX logo are not subject to the Creative Commons license and may not be reproduced without the prior and express written consent of Rice University.

Back to blog home

What's the formula for a significance test, the statsig team.

Ever wondered why some changes make a big splash while others don't seem to ripple? It often boils down to the backbone of decision-making in data analysis: statistical significance. This concept isn't just about crunching numbers; it's about making sure those numbers tell a true story, free from the distortion of random chance.

Statistical significance acts as a gatekeeper, ensuring that the decisions you make are based on solid evidence. Whether you're a business leader, a researcher, or a policymaker, understanding this concept can dramatically enhance the reliability of your conclusions and help you implement strategies that are truly effective.

Understanding statistical significance

Statistical significance is a critical concept in hypothesis testing. It helps you determine the probability that the observed difference between groups is not due to random chance. Here’s how it works:

Defining the role : In hypothesis testing, you start with a null hypothesis that assumes no effect or no difference between groups. Statistical significance tests whether the data you collect supports this assumption or if, conversely, you have enough evidence to reject it in favor of an alternative hypothesis.

The importance of statistical significance extends far beyond academic exercises; it plays a pivotal role in real-world decision making:

Informed decisions : Businesses use statistical significance to validate everything from marketing strategies to operational changes. By ensuring that results are not due to random fluctuations, leaders can confidently make decisions that are likely to result in positive outcomes.

Avoiding errors : Researchers and policymakers rely on statistical significance to avoid erroneous conclusions. This rigorous checking guards against the costly mistakes that might arise from acting on spurious data, ensuring that resources are directed in a manner that is truly beneficial.

In essence, understanding and applying statistical significance helps secure the integrity of your decisions, ensuring they are backed by evidence that is both reliable and replicable.

Key components of a significance test

Let's dive into the core elements of a significance test: the null and alternative hypotheses, and the pivotal roles of p-values and confidence intervals.

Null vs. alternative hypothesis : The null hypothesis (often symbolized as H0) suggests no effect or difference exists between the groups being tested. In contrast, the alternative hypothesis (H1) proposes that there is an effect or a difference. Understanding these opposing hypotheses helps you frame your experiment and anticipate different outcomes.

P-values and confidence intervals are cornerstone metrics in the realm of statistical testing. They offer nuanced insights into your data:

P-values : This metric helps you gauge the strength of the evidence against the null hypothesis. A small p-value (typically less than 0.05) suggests strong evidence against H0, indicating that your observed effect is unlikely due to chance alone.

Confidence intervals : These provide a range of values within which the true effect size likely falls. Broadly, they offer a snapshot of the data's reliability, helping you understand not just if an effect exists, but also its potential magnitude and relevance.

By mastering these components, you equip yourself with the analytical tools to make informed, data-driven decisions. These decisions are crucial whether you're refining a tech product, optimizing a marketing strategy, or influencing policy changes. For a deeper understanding of how these components function in real-world applications, consider exploring detailed examples and further explanations offered in resources like this comprehensive guide .

Calculating statistical significance

Let's break down the calculation of statistical significance into understanding common tests and a practical guide for calculations.

Overview of common statistical tests : You'll encounter several types of statistical tests, each suited for different data types and study designs. The t-test is ideal for comparing the means of two groups. For categorical data, the chi-square test assesses differences between groups. When dealing with more than two groups or variables, ANOVA is your go-to test, helping determine if there are any statistically significant differences between the means of three or more independent groups.

Step-by-step calculation guide : To calculate p-values and confidence intervals, you can either use statistical software or manual formulas. Begin with defining your null and alternative hypotheses. Select the appropriate test (t-test, chi-square, ANOVA) based on your data and research design. Input your data into the software or use the formula for the selected test. The software or the result from the formula will provide the p-value and, in many cases, the confidence interval. Remember, a p-value less than 0.05 typically suggests significant results against the null hypothesis.

By following these steps, you ensure that your findings on statistical significance are both reliable and valid, paving the way for informed decision-making based on your data.

Real-world applications of significance tests

In marketing and product development, A/B testing is a common tool. Statistical significance guides decisions by comparing campaign strategies. It ensures that changes in conversion rates are due to strategy, not chance. For a deeper understanding of how to enhance A/B testing strategies, consider exploring resources like Advanced Statistical Techniques in A/B Testing .

In medical research, significance tests are indispensable. They assess the effectiveness of new treatments in clinical trials. This process confirms whether observed benefits are statistically valid. For more insights on the application of significance tests in clinical settings, reviewing materials on Statistical Methods in Medical Research can be beneficial.

Common misinterpretations and best practices

Misunderstandings about p-values are common. Many believe a p-value indicates the probability that the null hypothesis is true; however, it simply measures the evidence against the null hypothesis. It's critical to grasp that a p-value does not confirm the null hypothesis's truth.

Best practices for reporting and interpretation are essential. Always report p-values alongside confidence intervals to provide a fuller picture of the data. This approach prevents overstating results and helps others understand the effect size and reliability of your findings.

Follow these guidelines to enhance the credibility of your research:

State the significance level explicitly when you report p-values.

Clarify the context of your hypothesis tests to avoid misinterpretation.

Discuss limitations of the p-value, such as its dependency on sample size.

By adhering to these practices, you ensure that your findings are both responsible and clear. This fosters a better understanding and application of statistical tests in various fields.

Actionable intelligence at your fingertips

With Statsig Analytics you can get answers in just a few clicks. No queries required.

Build fast?

Try statsig today.

Recent Posts

Statsig's eurotrip: a/b talks roadshow highlights.

Statsig Eurotrip: A/B Talks Roadshow with leaders from Monzo, HelloFresh, N26, Captify, Bell Statistics, and Babbel. Highlights and recordings inside!

Announcing the new suite of Statsig Javascript SDKs

Introducing @statsig/js-client: Our new JavaScript SDKs reduce package sizes by 60%, support web analytics and session replay, and simplify initialization.

How to Export Experimentation Results

Ensure your experiment results resonate with all stakeholders. Learn to present data effectively for both tech-savvy and business-oriented team members with this step-by-step guide.

Statsig's Autotune adds Contextual Bandits for personalization

Discover Statsig's Contextual Bandits in Autotune: a lightweight reinforcement learning tool for personalized user experiences and optimized results.

Warehouse Native Year in Review

Warehouse Native by Statsig brings real-time experimentation to customer data warehouses. Learn how it became a core product and what’s next for us.

Infrastructure testing in feature release process

Explore the importance of infrastructure testing in the feature release process. Ensure smooth deployments and robust performance with insights from Statsig.

User Preferences

Content preview.

Arcu felis bibendum ut tristique et egestas quis:

• Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris
• Duis aute irure dolor in reprehenderit in voluptate
• Excepteur sint occaecat cupidatat non proident

Keyboard Shortcuts

6a.1 - introduction to hypothesis testing, basic terms section  .

The first step in hypothesis testing is to set up two competing hypotheses. The hypotheses are the most important aspect. If the hypotheses are incorrect, your conclusion will also be incorrect.

The two hypotheses are named the null hypothesis and the alternative hypothesis.

The goal of hypothesis testing is to see if there is enough evidence against the null hypothesis. In other words, to see if there is enough evidence to reject the null hypothesis. If there is not enough evidence, then we fail to reject the null hypothesis.

Consider the following example where we set up these hypotheses.

Example 6-1 Section  

A man, Mr. Orangejuice, goes to trial and is tried for the murder of his ex-wife. He is either guilty or innocent. Set up the null and alternative hypotheses for this example.

Putting this in a hypothesis testing framework, the hypotheses being tested are:

• The man is guilty
• The man is innocent

Let's set up the null and alternative hypotheses.

\(H_0\colon \) Mr. Orangejuice is innocent

\(H_a\colon \) Mr. Orangejuice is guilty

Remember that we assume the null hypothesis is true and try to see if we have evidence against the null. Therefore, it makes sense in this example to assume the man is innocent and test to see if there is evidence that he is guilty.

The Logic of Hypothesis Testing Section  

We want to know the answer to a research question. We determine our null and alternative hypotheses. Now it is time to make a decision.

The decision is either going to be...

• reject the null hypothesis or...
• fail to reject the null hypothesis.

Consider the following table. The table shows the decision/conclusion of the hypothesis test and the unknown "reality", or truth. We do not know if the null is true or if it is false. If the null is false and we reject it, then we made the correct decision. If the null hypothesis is true and we fail to reject it, then we made the correct decision.

So what happens when we do not make the correct decision?

When doing hypothesis testing, two types of mistakes may be made and we call them Type I error and Type II error. If we reject the null hypothesis when it is true, then we made a type I error. If the null hypothesis is false and we failed to reject it, we made another error called a Type II error.

Types of errors

The “reality”, or truth, about the null hypothesis is unknown and therefore we do not know if we have made the correct decision or if we committed an error. We can, however, define the likelihood of these events.

\(\alpha\) and \(\beta\) are probabilities of committing an error so we want these values to be low. However, we cannot decrease both. As \(\alpha\) decreases, \(\beta\) increases.

Example 6-1 Cont'd... Section  

A man, Mr. Orangejuice, goes to trial and is tried for the murder of his ex-wife. He is either guilty or not guilty. We found before that...

• \( H_0\colon \) Mr. Orangejuice is innocent
• \( H_a\colon \) Mr. Orangejuice is guilty

Interpret Type I error, \(\alpha \), Type II error, \(\beta \).

As you can see here, the Type I error (putting an innocent man in jail) is the more serious error. Ethically, it is more serious to put an innocent man in jail than to let a guilty man go free. So to minimize the probability of a type I error we would choose a smaller significance level.

Try it! Section  

An inspector has to choose between certifying a building as safe or saying that the building is not safe. There are two hypotheses:

• Building is safe
• Building is not safe

Set up the null and alternative hypotheses. Interpret Type I and Type II error.

\( H_0\colon\) Building is not safe vs \(H_a\colon \) Building is safe

Power and \(\beta \) are complements of each other. Therefore, they have an inverse relationship, i.e. as one increases, the other decreases.

Warning: The NCBI web site requires JavaScript to function. more...

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing; 2024 Jan-.

StatPearls [Internet].

Hypothesis testing, p values, confidence intervals, and significance.

Jacob Shreffler ; Martin R. Huecker .

Affiliations

Last Update: March 13, 2023 .

• Definition/Introduction

Medical providers often rely on evidence-based medicine to guide decision-making in practice. Often a research hypothesis is tested with results provided, typically with p values, confidence intervals, or both. Additionally, statistical or research significance is estimated or determined by the investigators. Unfortunately, healthcare providers may have different comfort levels in interpreting these findings, which may affect the adequate application of the data.

• Issues of Concern

Without a foundational understanding of hypothesis testing, p values, confidence intervals, and the difference between statistical and clinical significance, it may affect healthcare providers' ability to make clinical decisions without relying purely on the research investigators deemed level of significance. Therefore, an overview of these concepts is provided to allow medical professionals to use their expertise to determine if results are reported sufficiently and if the study outcomes are clinically appropriate to be applied in healthcare practice.

Hypothesis Testing

Investigators conducting studies need research questions and hypotheses to guide analyses. Starting with broad research questions (RQs), investigators then identify a gap in current clinical practice or research. Any research problem or statement is grounded in a better understanding of relationships between two or more variables. For this article, we will use the following research question example:

Research Question: Is Drug 23 an effective treatment for Disease A?

Research questions do not directly imply specific guesses or predictions; we must formulate research hypotheses. A hypothesis is a predetermined declaration regarding the research question in which the investigator(s) makes a precise, educated guess about a study outcome. This is sometimes called the alternative hypothesis and ultimately allows the researcher to take a stance based on experience or insight from medical literature. An example of a hypothesis is below.

Research Hypothesis: Drug 23 will significantly reduce symptoms associated with Disease A compared to Drug 22.

The null hypothesis states that there is no statistical difference between groups based on the stated research hypothesis.

Researchers should be aware of journal recommendations when considering how to report p values, and manuscripts should remain internally consistent.

Regarding p values, as the number of individuals enrolled in a study (the sample size) increases, the likelihood of finding a statistically significant effect increases. With very large sample sizes, the p-value can be very low significant differences in the reduction of symptoms for Disease A between Drug 23 and Drug 22. The null hypothesis is deemed true until a study presents significant data to support rejecting the null hypothesis. Based on the results, the investigators will either reject the null hypothesis (if they found significant differences or associations) or fail to reject the null hypothesis (they could not provide proof that there were significant differences or associations).

To test a hypothesis, researchers obtain data on a representative sample to determine whether to reject or fail to reject a null hypothesis. In most research studies, it is not feasible to obtain data for an entire population. Using a sampling procedure allows for statistical inference, though this involves a certain possibility of error. [1]  When determining whether to reject or fail to reject the null hypothesis, mistakes can be made: Type I and Type II errors. Though it is impossible to ensure that these errors have not occurred, researchers should limit the possibilities of these faults. [2]

Significance

Significance is a term to describe the substantive importance of medical research. Statistical significance is the likelihood of results due to chance. [3]  Healthcare providers should always delineate statistical significance from clinical significance, a common error when reviewing biomedical research. [4]  When conceptualizing findings reported as either significant or not significant, healthcare providers should not simply accept researchers' results or conclusions without considering the clinical significance. Healthcare professionals should consider the clinical importance of findings and understand both p values and confidence intervals so they do not have to rely on the researchers to determine the level of significance. [5]  One criterion often used to determine statistical significance is the utilization of p values.

P values are used in research to determine whether the sample estimate is significantly different from a hypothesized value. The p-value is the probability that the observed effect within the study would have occurred by chance if, in reality, there was no true effect. Conventionally, data yielding a p<0.05 or p<0.01 is considered statistically significant. While some have debated that the 0.05 level should be lowered, it is still universally practiced. [6]  Hypothesis testing allows us to determine the size of the effect.

An example of findings reported with p values are below:

Statement: Drug 23 reduced patients' symptoms compared to Drug 22. Patients who received Drug 23 (n=100) were 2.1 times less likely than patients who received Drug 22 (n = 100) to experience symptoms of Disease A, p<0.05.

Statement:Individuals who were prescribed Drug 23 experienced fewer symptoms (M = 1.3, SD = 0.7) compared to individuals who were prescribed Drug 22 (M = 5.3, SD = 1.9). This finding was statistically significant, p= 0.02.

For either statement, if the threshold had been set at 0.05, the null hypothesis (that there was no relationship) should be rejected, and we should conclude significant differences. Noticeably, as can be seen in the two statements above, some researchers will report findings with < or > and others will provide an exact p-value (0.000001) but never zero [6] . When examining research, readers should understand how p values are reported. The best practice is to report all p values for all variables within a study design, rather than only providing p values for variables with significant findings. [7]  The inclusion of all p values provides evidence for study validity and limits suspicion for selective reporting/data mining.  

While researchers have historically used p values, experts who find p values problematic encourage the use of confidence intervals. [8] . P-values alone do not allow us to understand the size or the extent of the differences or associations. [3]  In March 2016, the American Statistical Association (ASA) released a statement on p values, noting that scientific decision-making and conclusions should not be based on a fixed p-value threshold (e.g., 0.05). They recommend focusing on the significance of results in the context of study design, quality of measurements, and validity of data. Ultimately, the ASA statement noted that in isolation, a p-value does not provide strong evidence. [9]

When conceptualizing clinical work, healthcare professionals should consider p values with a concurrent appraisal study design validity. For example, a p-value from a double-blinded randomized clinical trial (designed to minimize bias) should be weighted higher than one from a retrospective observational study [7] . The p-value debate has smoldered since the 1950s [10] , and replacement with confidence intervals has been suggested since the 1980s. [11]

Confidence Intervals

A confidence interval provides a range of values within given confidence (e.g., 95%), including the accurate value of the statistical constraint within a targeted population. [12]  Most research uses a 95% CI, but investigators can set any level (e.g., 90% CI, 99% CI). [13]  A CI provides a range with the lower bound and upper bound limits of a difference or association that would be plausible for a population. [14]  Therefore, a CI of 95% indicates that if a study were to be carried out 100 times, the range would contain the true value in 95, [15]  confidence intervals provide more evidence regarding the precision of an estimate compared to p-values. [6]

In consideration of the similar research example provided above, one could make the following statement with 95% CI:

Statement: Individuals who were prescribed Drug 23 had no symptoms after three days, which was significantly faster than those prescribed Drug 22; there was a mean difference between the two groups of days to the recovery of 4.2 days (95% CI: 1.9 – 7.8).

It is important to note that the width of the CI is affected by the standard error and the sample size; reducing a study sample number will result in less precision of the CI (increase the width). [14]  A larger width indicates a smaller sample size or a larger variability. [16]  A researcher would want to increase the precision of the CI. For example, a 95% CI of 1.43 – 1.47 is much more precise than the one provided in the example above. In research and clinical practice, CIs provide valuable information on whether the interval includes or excludes any clinically significant values. [14]

Null values are sometimes used for differences with CI (zero for differential comparisons and 1 for ratios). However, CIs provide more information than that. [15]  Consider this example: A hospital implements a new protocol that reduced wait time for patients in the emergency department by an average of 25 minutes (95% CI: -2.5 – 41 minutes). Because the range crosses zero, implementing this protocol in different populations could result in longer wait times; however, the range is much higher on the positive side. Thus, while the p-value used to detect statistical significance for this may result in "not significant" findings, individuals should examine this range, consider the study design, and weigh whether or not it is still worth piloting in their workplace.

Similarly to p-values, 95% CIs cannot control for researchers' errors (e.g., study bias or improper data analysis). [14]  In consideration of whether to report p-values or CIs, researchers should examine journal preferences. When in doubt, reporting both may be beneficial. [13]  An example is below:

Reporting both: Individuals who were prescribed Drug 23 had no symptoms after three days, which was significantly faster than those prescribed Drug 22, p = 0.009. There was a mean difference between the two groups of days to the recovery of 4.2 days (95% CI: 1.9 – 7.8).

• Clinical Significance

Recall that clinical significance and statistical significance are two different concepts. Healthcare providers should remember that a study with statistically significant differences and large sample size may be of no interest to clinicians, whereas a study with smaller sample size and statistically non-significant results could impact clinical practice. [14]  Additionally, as previously mentioned, a non-significant finding may reflect the study design itself rather than relationships between variables.

Healthcare providers using evidence-based medicine to inform practice should use clinical judgment to determine the practical importance of studies through careful evaluation of the design, sample size, power, likelihood of type I and type II errors, data analysis, and reporting of statistical findings (p values, 95% CI or both). [4]  Interestingly, some experts have called for "statistically significant" or "not significant" to be excluded from work as statistical significance never has and will never be equivalent to clinical significance. [17]

The decision on what is clinically significant can be challenging, depending on the providers' experience and especially the severity of the disease. Providers should use their knowledge and experiences to determine the meaningfulness of study results and make inferences based not only on significant or insignificant results by researchers but through their understanding of study limitations and practical implications.

• Nursing, Allied Health, and Interprofessional Team Interventions

All physicians, nurses, pharmacists, and other healthcare professionals should strive to understand the concepts in this chapter. These individuals should maintain the ability to review and incorporate new literature for evidence-based and safe care. 

• Review Questions
• Access free multiple choice questions on this topic.
• Comment on this article.

Disclosure: Jacob Shreffler declares no relevant financial relationships with ineligible companies.

Disclosure: Martin Huecker declares no relevant financial relationships with ineligible companies.

This book is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) ( http://creativecommons.org/licenses/by-nc-nd/4.0/ ), which permits others to distribute the work, provided that the article is not altered or used commercially. You are not required to obtain permission to distribute this article, provided that you credit the author and journal.

• Cite this Page Shreffler J, Huecker MR. Hypothesis Testing, P Values, Confidence Intervals, and Significance. [Updated 2023 Mar 13]. In: StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing; 2024 Jan-.

In this Page

Bulk download.

• Bulk download StatPearls data from FTP

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Similar articles in PubMed

• The reporting of p values, confidence intervals and statistical significance in Preventive Veterinary Medicine (1997-2017). [PeerJ. 2021] The reporting of p values, confidence intervals and statistical significance in Preventive Veterinary Medicine (1997-2017). Messam LLM, Weng HY, Rosenberger NWY, Tan ZH, Payet SDM, Santbakshsing M. PeerJ. 2021; 9:e12453. Epub 2021 Nov 24.
• Review Clinical versus statistical significance: interpreting P values and confidence intervals related to measures of association to guide decision making. [J Pharm Pract. 2010] Review Clinical versus statistical significance: interpreting P values and confidence intervals related to measures of association to guide decision making. Ferrill MJ, Brown DA, Kyle JA. J Pharm Pract. 2010 Aug; 23(4):344-51. Epub 2010 Apr 13.
• Interpreting "statistical hypothesis testing" results in clinical research. [J Ayurveda Integr Med. 2012] Interpreting "statistical hypothesis testing" results in clinical research. Sarmukaddam SB. J Ayurveda Integr Med. 2012 Apr; 3(2):65-9.
• Confidence intervals in procedural dermatology: an intuitive approach to interpreting data. [Dermatol Surg. 2005] Confidence intervals in procedural dermatology: an intuitive approach to interpreting data. Alam M, Barzilai DA, Wrone DA. Dermatol Surg. 2005 Apr; 31(4):462-6.
• Review Is statistical significance testing useful in interpreting data? [Reprod Toxicol. 1993] Review Is statistical significance testing useful in interpreting data? Savitz DA. Reprod Toxicol. 1993; 7(2):95-100.

Recent Activity

• Hypothesis Testing, P Values, Confidence Intervals, and Significance - StatPearl... Hypothesis Testing, P Values, Confidence Intervals, and Significance - StatPearls

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Hypothesis Testing | A Step-by-Step Guide with Easy Examples

Published on November 8, 2019 by Rebecca Bevans . Revised on June 22, 2023.

Hypothesis testing is a formal procedure for investigating our ideas about the world using statistics . It is most often used by scientists to test specific predictions, called hypotheses, that arise from theories.

There are 5 main steps in hypothesis testing:

• State your research hypothesis as a null hypothesis and alternate hypothesis (H o ) and (H a  or H 1 ).
• Collect data in a way designed to test the hypothesis.
• Perform an appropriate statistical test .
• Decide whether to reject or fail to reject your null hypothesis.
• Present the findings in your results and discussion section.

Though the specific details might vary, the procedure you will use when testing a hypothesis will always follow some version of these steps.

Table of contents

Step 1: state your null and alternate hypothesis, step 2: collect data, step 3: perform a statistical test, step 4: decide whether to reject or fail to reject your null hypothesis, step 5: present your findings, other interesting articles, frequently asked questions about hypothesis testing.

After developing your initial research hypothesis (the prediction that you want to investigate), it is important to restate it as a null (H o ) and alternate (H a ) hypothesis so that you can test it mathematically.

The alternate hypothesis is usually your initial hypothesis that predicts a relationship between variables. The null hypothesis is a prediction of no relationship between the variables you are interested in.

• H 0 : Men are, on average, not taller than women. H a : Men are, on average, taller than women.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

For a statistical test to be valid , it is important to perform sampling and collect data in a way that is designed to test your hypothesis. If your data are not representative, then you cannot make statistical inferences about the population you are interested in.

There are a variety of statistical tests available, but they are all based on the comparison of within-group variance (how spread out the data is within a category) versus between-group variance (how different the categories are from one another).

If the between-group variance is large enough that there is little or no overlap between groups, then your statistical test will reflect that by showing a low p -value . This means it is unlikely that the differences between these groups came about by chance.

Alternatively, if there is high within-group variance and low between-group variance, then your statistical test will reflect that with a high p -value. This means it is likely that any difference you measure between groups is due to chance.

Your choice of statistical test will be based on the type of variables and the level of measurement of your collected data .

• an estimate of the difference in average height between the two groups.
• a p -value showing how likely you are to see this difference if the null hypothesis of no difference is true.

Based on the outcome of your statistical test, you will have to decide whether to reject or fail to reject your null hypothesis.

In most cases you will use the p -value generated by your statistical test to guide your decision. And in most cases, your predetermined level of significance for rejecting the null hypothesis will be 0.05 – that is, when there is a less than 5% chance that you would see these results if the null hypothesis were true.

In some cases, researchers choose a more conservative level of significance, such as 0.01 (1%). This minimizes the risk of incorrectly rejecting the null hypothesis ( Type I error ).

Prevent plagiarism. Run a free check.

The results of hypothesis testing will be presented in the results and discussion sections of your research paper , dissertation or thesis .

In the results section you should give a brief summary of the data and a summary of the results of your statistical test (for example, the estimated difference between group means and associated p -value). In the discussion , you can discuss whether your initial hypothesis was supported by your results or not.

In the formal language of hypothesis testing, we talk about rejecting or failing to reject the null hypothesis. You will probably be asked to do this in your statistics assignments.

However, when presenting research results in academic papers we rarely talk this way. Instead, we go back to our alternate hypothesis (in this case, the hypothesis that men are on average taller than women) and state whether the result of our test did or did not support the alternate hypothesis.

If your null hypothesis was rejected, this result is interpreted as “supported the alternate hypothesis.”

These are superficial differences; you can see that they mean the same thing.

You might notice that we don’t say that we reject or fail to reject the alternate hypothesis . This is because hypothesis testing is not designed to prove or disprove anything. It is only designed to test whether a pattern we measure could have arisen spuriously, or by chance.

If we reject the null hypothesis based on our research (i.e., we find that it is unlikely that the pattern arose by chance), then we can say our test lends support to our hypothesis . But if the pattern does not pass our decision rule, meaning that it could have arisen by chance, then we say the test is inconsistent with our hypothesis .

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Normal distribution
• Descriptive statistics
• Measures of central tendency
• Correlation coefficient

Methodology

• Cluster sampling
• Stratified sampling
• Types of interviews
• Cohort study
• Thematic analysis

Research bias

• Implicit bias
• Cognitive bias
• Survivorship bias
• Availability heuristic
• Nonresponse bias
• Regression to the mean

Hypothesis testing is a formal procedure for investigating our ideas about the world using statistics. It is used by scientists to test specific predictions, called hypotheses , by calculating how likely it is that a pattern or relationship between variables could have arisen by chance.

A hypothesis states your predictions about what your research will find. It is a tentative answer to your research question that has not yet been tested. For some research projects, you might have to write several hypotheses that address different aspects of your research question.

A hypothesis is not just a guess — it should be based on existing theories and knowledge. It also has to be testable, which means you can support or refute it through scientific research methods (such as experiments, observations and statistical analysis of data).

Null and alternative hypotheses are used in statistical hypothesis testing . The null hypothesis of a test always predicts no effect or no relationship between variables, while the alternative hypothesis states your research prediction of an effect or relationship.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Bevans, R. (2023, June 22). Hypothesis Testing | A Step-by-Step Guide with Easy Examples. Scribbr. Retrieved July 1, 2024, from https://www.scribbr.com/statistics/hypothesis-testing/

Is this article helpful?

Rebecca Bevans

Other students also liked, choosing the right statistical test | types & examples, understanding p values | definition and examples, what is your plagiarism score.

P-Value And Statistical Significance: What It Is & Why It Matters

Saul Mcleod, PhD

Editor-in-Chief for Simply Psychology

BSc (Hons) Psychology, MRes, PhD, University of Manchester

Saul Mcleod, PhD., is a qualified psychology teacher with over 18 years of experience in further and higher education. He has been published in peer-reviewed journals, including the Journal of Clinical Psychology.

Learn about our Editorial Process

Olivia Guy-Evans, MSc

Associate Editor for Simply Psychology

BSc (Hons) Psychology, MSc Psychology of Education

Olivia Guy-Evans is a writer and associate editor for Simply Psychology. She has previously worked in healthcare and educational sectors.

On This Page:

The p-value in statistics quantifies the evidence against a null hypothesis. A low p-value suggests data is inconsistent with the null, potentially favoring an alternative hypothesis. Common significance thresholds are 0.05 or 0.01.

Hypothesis testing

When you perform a statistical test, a p-value helps you determine the significance of your results in relation to the null hypothesis.

The null hypothesis (H0) states no relationship exists between the two variables being studied (one variable does not affect the other). It states the results are due to chance and are not significant in supporting the idea being investigated. Thus, the null hypothesis assumes that whatever you try to prove did not happen.

The alternative hypothesis (Ha or H1) is the one you would believe if the null hypothesis is concluded to be untrue.

The alternative hypothesis states that the independent variable affected the dependent variable, and the results are significant in supporting the theory being investigated (i.e., the results are not due to random chance).

What a p-value tells you

A p-value, or probability value, is a number describing how likely it is that your data would have occurred by random chance (i.e., that the null hypothesis is true).

The level of statistical significance is often expressed as a p-value between 0 and 1.

The smaller the p -value, the less likely the results occurred by random chance, and the stronger the evidence that you should reject the null hypothesis.

Remember, a p-value doesn’t tell you if the null hypothesis is true or false. It just tells you how likely you’d see the data you observed (or more extreme data) if the null hypothesis was true. It’s a piece of evidence, not a definitive proof.

Example: Test Statistic and p-Value

Suppose you’re conducting a study to determine whether a new drug has an effect on pain relief compared to a placebo. If the new drug has no impact, your test statistic will be close to the one predicted by the null hypothesis (no difference between the drug and placebo groups), and the resulting p-value will be close to 1. It may not be precisely 1 because real-world variations may exist. Conversely, if the new drug indeed reduces pain significantly, your test statistic will diverge further from what’s expected under the null hypothesis, and the p-value will decrease. The p-value will never reach zero because there’s always a slim possibility, though highly improbable, that the observed results occurred by random chance.

P-value interpretation

The significance level (alpha) is a set probability threshold (often 0.05), while the p-value is the probability you calculate based on your study or analysis.

A p-value less than or equal to your significance level (typically ≤ 0.05) is statistically significant.

A p-value less than or equal to a predetermined significance level (often 0.05 or 0.01) indicates a statistically significant result, meaning the observed data provide strong evidence against the null hypothesis.

This suggests the effect under study likely represents a real relationship rather than just random chance.

For instance, if you set α = 0.05, you would reject the null hypothesis if your p -value ≤ 0.05. 

It indicates strong evidence against the null hypothesis, as there is less than a 5% probability the null is correct (and the results are random).

Therefore, we reject the null hypothesis and accept the alternative hypothesis.

Example: Statistical Significance

Upon analyzing the pain relief effects of the new drug compared to the placebo, the computed p-value is less than 0.01, which falls well below the predetermined alpha value of 0.05. Consequently, you conclude that there is a statistically significant difference in pain relief between the new drug and the placebo.

What does a p-value of 0.001 mean?

A p-value of 0.001 is highly statistically significant beyond the commonly used 0.05 threshold. It indicates strong evidence of a real effect or difference, rather than just random variation.

Specifically, a p-value of 0.001 means there is only a 0.1% chance of obtaining a result at least as extreme as the one observed, assuming the null hypothesis is correct.

Such a small p-value provides strong evidence against the null hypothesis, leading to rejecting the null in favor of the alternative hypothesis.

A p-value more than the significance level (typically p > 0.05) is not statistically significant and indicates strong evidence for the null hypothesis.

This means we retain the null hypothesis and reject the alternative hypothesis. You should note that you cannot accept the null hypothesis; we can only reject it or fail to reject it.

Note : when the p-value is above your threshold of significance,  it does not mean that there is a 95% probability that the alternative hypothesis is true.

One-Tailed Test

Two-Tailed Test

How do you calculate the p-value ?

Most statistical software packages like R, SPSS, and others automatically calculate your p-value. This is the easiest and most common way.

Online resources and tables are available to estimate the p-value based on your test statistic and degrees of freedom.

These tables help you understand how often you would expect to see your test statistic under the null hypothesis.

Understanding the Statistical Test:

Different statistical tests are designed to answer specific research questions or hypotheses. Each test has its own underlying assumptions and characteristics.

For example, you might use a t-test to compare means, a chi-squared test for categorical data, or a correlation test to measure the strength of a relationship between variables.

Be aware that the number of independent variables you include in your analysis can influence the magnitude of the test statistic needed to produce the same p-value.

This factor is particularly important to consider when comparing results across different analyses.

Example: Choosing a Statistical Test

If you’re comparing the effectiveness of just two different drugs in pain relief, a two-sample t-test is a suitable choice for comparing these two groups. However, when you’re examining the impact of three or more drugs, it’s more appropriate to employ an Analysis of Variance ( ANOVA) . Utilizing multiple pairwise comparisons in such cases can lead to artificially low p-values and an overestimation of the significance of differences between the drug groups.

How to report

A statistically significant result cannot prove that a research hypothesis is correct (which implies 100% certainty).

Instead, we may state our results “provide support for” or “give evidence for” our research hypothesis (as there is still a slight probability that the results occurred by chance and the null hypothesis was correct – e.g., less than 5%).

Example: Reporting the results

In our comparison of the pain relief effects of the new drug and the placebo, we observed that participants in the drug group experienced a significant reduction in pain ( M = 3.5; SD = 0.8) compared to those in the placebo group ( M = 5.2; SD  = 0.7), resulting in an average difference of 1.7 points on the pain scale (t(98) = -9.36; p < 0.001).

The 6th edition of the APA style manual (American Psychological Association, 2010) states the following on the topic of reporting p-values:

“When reporting p values, report exact p values (e.g., p = .031) to two or three decimal places. However, report p values less than .001 as p < .001.

The tradition of reporting p values in the form p < .10, p < .05, p < .01, and so forth, was appropriate in a time when only limited tables of critical values were available.” (p. 114)

• Do not use 0 before the decimal point for the statistical value p as it cannot equal 1. In other words, write p = .001 instead of p = 0.001.
• Please pay attention to issues of italics ( p is always italicized) and spacing (either side of the = sign).
• p = .000 (as outputted by some statistical packages such as SPSS) is impossible and should be written as p < .001.
• The opposite of significant is “nonsignificant,” not “insignificant.”

Why is the p -value not enough?

A lower p-value  is sometimes interpreted as meaning there is a stronger relationship between two variables.

However, statistical significance means that it is unlikely that the null hypothesis is true (less than 5%).

To understand the strength of the difference between the two groups (control vs. experimental) a researcher needs to calculate the effect size .

When do you reject the null hypothesis?

In statistical hypothesis testing, you reject the null hypothesis when the p-value is less than or equal to the significance level (α) you set before conducting your test. The significance level is the probability of rejecting the null hypothesis when it is true. Commonly used significance levels are 0.01, 0.05, and 0.10.

Remember, rejecting the null hypothesis doesn’t prove the alternative hypothesis; it just suggests that the alternative hypothesis may be plausible given the observed data.

The p -value is conditional upon the null hypothesis being true but is unrelated to the truth or falsity of the alternative hypothesis.

What does p-value of 0.05 mean?

If your p-value is less than or equal to 0.05 (the significance level), you would conclude that your result is statistically significant. This means the evidence is strong enough to reject the null hypothesis in favor of the alternative hypothesis.

Are all p-values below 0.05 considered statistically significant?

No, not all p-values below 0.05 are considered statistically significant. The threshold of 0.05 is commonly used, but it’s just a convention. Statistical significance depends on factors like the study design, sample size, and the magnitude of the observed effect.

A p-value below 0.05 means there is evidence against the null hypothesis, suggesting a real effect. However, it’s essential to consider the context and other factors when interpreting results.

Researchers also look at effect size and confidence intervals to determine the practical significance and reliability of findings.

How does sample size affect the interpretation of p-values?

Sample size can impact the interpretation of p-values. A larger sample size provides more reliable and precise estimates of the population, leading to narrower confidence intervals.

With a larger sample, even small differences between groups or effects can become statistically significant, yielding lower p-values. In contrast, smaller sample sizes may not have enough statistical power to detect smaller effects, resulting in higher p-values.

Therefore, a larger sample size increases the chances of finding statistically significant results when there is a genuine effect, making the findings more trustworthy and robust.

Can a non-significant p-value indicate that there is no effect or difference in the data?

No, a non-significant p-value does not necessarily indicate that there is no effect or difference in the data. It means that the observed data do not provide strong enough evidence to reject the null hypothesis.

There could still be a real effect or difference, but it might be smaller or more variable than the study was able to detect.

Other factors like sample size, study design, and measurement precision can influence the p-value. It’s important to consider the entire body of evidence and not rely solely on p-values when interpreting research findings.

Can P values be exactly zero?

While a p-value can be extremely small, it cannot technically be absolute zero. When a p-value is reported as p = 0.000, the actual p-value is too small for the software to display. This is often interpreted as strong evidence against the null hypothesis. For p values less than 0.001, report as p < .001

Further Information

• P Value Calculator From T Score
• P-Value Calculator For Chi-Square
• P-values and significance tests (Kahn Academy)
• Hypothesis testing and p-values (Kahn Academy)
• Wasserstein, R. L., Schirm, A. L., & Lazar, N. A. (2019). Moving to a world beyond “ p “< 0.05”.
• Criticism of using the “ p “< 0.05”.
• Publication manual of the American Psychological Association
• Statistics for Psychology Book Download

Bland, J. M., & Altman, D. G. (1994). One and two sided tests of significance: Authors’ reply.  BMJ: British Medical Journal ,  309 (6958), 874.

Goodman, S. N., & Royall, R. (1988). Evidence and scientific research.  American Journal of Public Health ,  78 (12), 1568-1574.

Goodman, S. (2008, July). A dirty dozen: twelve p-value misconceptions . In  Seminars in hematology  (Vol. 45, No. 3, pp. 135-140). WB Saunders.

Lang, J. M., Rothman, K. J., & Cann, C. I. (1998). That confounded P-value.  Epidemiology (Cambridge, Mass.) ,  9 (1), 7-8.

• Skip to secondary menu
• Skip to main content
• Skip to primary sidebar

Statistics By Jim

Making statistics intuitive

Statistical Hypothesis Testing Overview

By Jim Frost 59 Comments

In this blog post, I explain why you need to use statistical hypothesis testing and help you navigate the essential terminology. Hypothesis testing is a crucial procedure to perform when you want to make inferences about a population using a random sample. These inferences include estimating population properties such as the mean, differences between means, proportions, and the relationships between variables.

This post provides an overview of statistical hypothesis testing. If you need to perform hypothesis tests, consider getting my book, Hypothesis Testing: An Intuitive Guide .

Why You Should Perform Statistical Hypothesis Testing

Hypothesis testing is a form of inferential statistics that allows us to draw conclusions about an entire population based on a representative sample. You gain tremendous benefits by working with a sample. In most cases, it is simply impossible to observe the entire population to understand its properties. The only alternative is to collect a random sample and then use statistics to analyze it.

While samples are much more practical and less expensive to work with, there are trade-offs. When you estimate the properties of a population from a sample, the sample statistics are unlikely to equal the actual population value exactly.  For instance, your sample mean is unlikely to equal the population mean. The difference between the sample statistic and the population value is the sample error.

Differences that researchers observe in samples might be due to sampling error rather than representing a true effect at the population level. If sampling error causes the observed difference, the next time someone performs the same experiment the results might be different. Hypothesis testing incorporates estimates of the sampling error to help you make the correct decision. Learn more about Sampling Error .

For example, if you are studying the proportion of defects produced by two manufacturing methods, any difference you observe between the two sample proportions might be sample error rather than a true difference. If the difference does not exist at the population level, you won’t obtain the benefits that you expect based on the sample statistics. That can be a costly mistake!

Let’s cover some basic hypothesis testing terms that you need to know.

Background information : Difference between Descriptive and Inferential Statistics and Populations, Parameters, and Samples in Inferential Statistics

Hypothesis Testing

Hypothesis testing is a statistical analysis that uses sample data to assess two mutually exclusive theories about the properties of a population. Statisticians call these theories the null hypothesis and the alternative hypothesis. A hypothesis test assesses your sample statistic and factors in an estimate of the sample error to determine which hypothesis the data support.

When you can reject the null hypothesis, the results are statistically significant, and your data support the theory that an effect exists at the population level.

The effect is the difference between the population value and the null hypothesis value. The effect is also known as population effect or the difference. For example, the mean difference between the health outcome for a treatment group and a control group is the effect.

Typically, you do not know the size of the actual effect. However, you can use a hypothesis test to help you determine whether an effect exists and to estimate its size. Hypothesis tests convert your sample effect into a test statistic, which it evaluates for statistical significance. Learn more about Test Statistics .

An effect can be statistically significant, but that doesn’t necessarily indicate that it is important in a real-world, practical sense. For more information, read my post about Statistical vs. Practical Significance .

Null Hypothesis

The null hypothesis is one of two mutually exclusive theories about the properties of the population in hypothesis testing. Typically, the null hypothesis states that there is no effect (i.e., the effect size equals zero). The null is often signified by H 0 .

In all hypothesis testing, the researchers are testing an effect of some sort. The effect can be the effectiveness of a new vaccination, the durability of a new product, the proportion of defect in a manufacturing process, and so on. There is some benefit or difference that the researchers hope to identify.

However, it’s possible that there is no effect or no difference between the experimental groups. In statistics, we call this lack of an effect the null hypothesis. Therefore, if you can reject the null, you can favor the alternative hypothesis, which states that the effect exists (doesn’t equal zero) at the population level.

You can think of the null as the default theory that requires sufficiently strong evidence against in order to reject it.

For example, in a 2-sample t-test, the null often states that the difference between the two means equals zero.

When you can reject the null hypothesis, your results are statistically significant. Learn more about Statistical Significance: Definition & Meaning .

Related post : Understanding the Null Hypothesis in More Detail

Alternative Hypothesis

The alternative hypothesis is the other theory about the properties of the population in hypothesis testing. Typically, the alternative hypothesis states that a population parameter does not equal the null hypothesis value. In other words, there is a non-zero effect. If your sample contains sufficient evidence, you can reject the null and favor the alternative hypothesis. The alternative is often identified with H 1 or H A .

For example, in a 2-sample t-test, the alternative often states that the difference between the two means does not equal zero.

You can specify either a one- or two-tailed alternative hypothesis:

If you perform a two-tailed hypothesis test, the alternative states that the population parameter does not equal the null value. For example, when the alternative hypothesis is H A : μ ≠ 0, the test can detect differences both greater than and less than the null value.

A one-tailed alternative has more power to detect an effect but it can test for a difference in only one direction. For example, H A : μ > 0 can only test for differences that are greater than zero.

Related posts : Understanding T-tests and One-Tailed and Two-Tailed Hypothesis Tests Explained

P-values are the probability that you would obtain the effect observed in your sample, or larger, if the null hypothesis is correct. In simpler terms, p-values tell you how strongly your sample data contradict the null. Lower p-values represent stronger evidence against the null. You use P-values in conjunction with the significance level to determine whether your data favor the null or alternative hypothesis.

Related post : Interpreting P-values Correctly

Significance Level (Alpha)

For instance, a significance level of 0.05 signifies a 5% risk of deciding that an effect exists when it does not exist.

Use p-values and significance levels together to help you determine which hypothesis the data support. If the p-value is less than your significance level, you can reject the null and conclude that the effect is statistically significant. In other words, the evidence in your sample is strong enough to be able to reject the null hypothesis at the population level.

Related posts : Graphical Approach to Significance Levels and P-values and Conceptual Approach to Understanding Significance Levels

Types of Errors in Hypothesis Testing

Statistical hypothesis tests are not 100% accurate because they use a random sample to draw conclusions about entire populations. There are two types of errors related to drawing an incorrect conclusion.

• False positives: You reject a null that is true. Statisticians call this a Type I error . The Type I error rate equals your significance level or alpha (α).
• False negatives: You fail to reject a null that is false. Statisticians call this a Type II error. Generally, you do not know the Type II error rate. However, it is a larger risk when you have a small sample size , noisy data, or a small effect size. The type II error rate is also known as beta (β).

Statistical power is the probability that a hypothesis test correctly infers that a sample effect exists in the population. In other words, the test correctly rejects a false null hypothesis. Consequently, power is inversely related to a Type II error. Power = 1 – β. Learn more about Power in Statistics .

Related posts : Types of Errors in Hypothesis Testing and Estimating a Good Sample Size for Your Study Using Power Analysis

Which Type of Hypothesis Test is Right for You?

There are many different types of procedures you can use. The correct choice depends on your research goals and the data you collect. Do you need to understand the mean or the differences between means? Or, perhaps you need to assess proportions. You can even use hypothesis testing to determine whether the relationships between variables are statistically significant.

To choose the proper statistical procedure, you’ll need to assess your study objectives and collect the correct type of data . This background research is necessary before you begin a study.

Related Post : Hypothesis Tests for Continuous, Binary, and Count Data

Statistical tests are crucial when you want to use sample data to make conclusions about a population because these tests account for sample error. Using significance levels and p-values to determine when to reject the null hypothesis improves the probability that you will draw the correct conclusion.

To see an alternative approach to these traditional hypothesis testing methods, learn about bootstrapping in statistics !

If you want to see examples of hypothesis testing in action, I recommend the following posts that I have written:

• How Effective Are Flu Shots? This example shows how you can use statistics to test proportions.
• Fatality Rates in Star Trek . This example shows how to use hypothesis testing with categorical data.
• Busting Myths About the Battle of the Sexes . A fun example based on a Mythbusters episode that assess continuous data using several different tests.
• Are Yawns Contagious? Another fun example inspired by a Mythbusters episode.

Share this:

Reader Interactions

January 14, 2024 at 8:43 am

Hello professor Jim, how are you doing! Pls. What are the properties of a population and their examples? Thanks for your time and understanding.

January 14, 2024 at 12:57 pm

Please read my post about Populations vs. Samples for more information and examples.

Also, please note there is a search bar in the upper-right margin of my website. Use that to search for topics.

July 5, 2023 at 7:05 am

Hello, I have a question as I read your post. You say in p-values section

“P-values are the probability that you would obtain the effect observed in your sample, or larger, if the null hypothesis is correct. In simpler terms, p-values tell you how strongly your sample data contradict the null. Lower p-values represent stronger evidence against the null.”

But according to your definition of effect, the null states that an effect does not exist, correct? So what I assume you want to say is that “P-values are the probability that you would obtain the effect observed in your sample, or larger, if the null hypothesis is **incorrect**.”

July 6, 2023 at 5:18 am

Hi Shrinivas,

The correct definition of p-value is that it is a probability that exists in the context of a true null hypothesis. So, the quotation is correct in stating “if the null hypothesis is correct.”

Essentially, the p-value tells you the likelihood of your observed results (or more extreme) if the null hypothesis is true. It gives you an idea of whether your results are surprising or unusual if there is no effect.

Hence, with sufficiently low p-values, you reject the null hypothesis because it’s telling you that your sample results were unlikely to have occurred if there was no effect in the population.

I hope that helps make it more clear. If not, let me know I’ll attempt to clarify!

May 8, 2023 at 12:47 am

Thanks a lot Ny best regards

May 7, 2023 at 11:15 pm

Hi Jim Can you tell me something about size effect? Thanks

May 8, 2023 at 12:29 am

Here’s a post that I’ve written about Effect Sizes that will hopefully tell you what you need to know. Please read that. Then, if you have any more specific questions about effect sizes, please post them there. Thanks!

January 7, 2023 at 4:19 pm

Hi Jim, I have only read two pages so far but I am really amazed because in few paragraphs you made me clearly understand the concepts of months of courses I received in biostatistics! Thanks so much for this work you have done it helps a lot!

January 10, 2023 at 3:25 pm

Thanks so much!

June 17, 2021 at 1:45 pm

Can you help in the following question: Rocinante36 is priced at ₹7 lakh and has been designed to deliver a mileage of 22 km/litre and a top speed of 140 km/hr. Formulate the null and alternative hypotheses for mileage and top speed to check whether the new models are performing as per the desired design specifications.

April 19, 2021 at 1:51 pm

Its indeed great to read your work statistics.

I have a doubt regarding the one sample t-test. So as per your book on hypothesis testing with reference to page no 45, you have mentioned the difference between “the sample mean and the hypothesised mean is statistically significant”. So as per my understanding it should be quoted like “the difference between the population mean and the hypothesised mean is statistically significant”. The catch here is the hypothesised mean represents the sample mean.

Please help me understand this.

Regards Rajat

April 19, 2021 at 3:46 pm

Thanks for buying my book. I’m so glad it’s been helpful!

The test is performed on the sample but the results apply to the population. Hence, if the difference between the sample mean (observed in your study) and the hypothesized mean is statistically significant, that suggests that population does not equal the hypothesized mean.

For one sample tests, the hypothesized mean is not the sample mean. It is a mean that you want to use for the test value. It usually represents a value that is important to your research. In other words, it’s a value that you pick for some theoretical/practical reasons. You pick it because you want to determine whether the population mean is different from that particular value.

I hope that helps!

November 5, 2020 at 6:24 am

Jim, you are such a magnificent statistician/economist/econometrician/data scientist etc whatever profession. Your work inspires and simplifies the lives of so many researchers around the world. I truly admire you and your work. I will buy a copy of each book you have on statistics or econometrics. Keep doing the good work. Remain ever blessed

November 6, 2020 at 9:47 pm

Hi Renatus,

Thanks so much for you very kind comments. You made my day!! I’m so glad that my website has been helpful. And, thanks so much for supporting my books! 🙂

November 2, 2020 at 9:32 pm

Hi Jim, I hope you are aware of 2019 American Statistical Association’s official statement on Statistical Significance: https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1583913 In case you do not bother reading the full article, may I quote you the core message here: “We conclude, based on our review of the articles in this special issue and the broader literature, that it is time to stop using the term “statistically significant” entirely. Nor should variants such as “significantly different,” “p < 0.05,” and “nonsignificant” survive, whether expressed in words, by asterisks in a table, or in some other way."

With best wishes,

November 3, 2020 at 2:09 am

I’m definitely aware of the debate surrounding how to use p-values most effectively. However, I need to correct you on one point. The link you provide is NOT a statement by the American Statistical Association. It is an editorial by several authors.

There is considerable debate over this issue. There are problems with p-values. However, as the authors state themselves, much of the problem is over people’s mindsets about how to use p-values and their incorrect interpretations about what statistical significance does and does not mean.

If you were to read my website more thoroughly, you’d be aware that I share many of their concerns and I address them in multiple posts. One of the authors’ key points is the need to be thoughtful and conduct thoughtful research and analysis. I emphasize this aspect in multiple posts on this topic. I’ll ask you to read the following three because they all address some of the authors’ concerns and suggestions. But you might run across others to read as well.

Five Tips for Using P-values to Avoid Being Misled How to Interpret P-values Correctly P-values and the Reproducibility of Experimental Results

September 24, 2020 at 11:52 pm

HI Jim, i just want you to know that you made explanation for Statistics so simple! I should say lesser and fewer words that reduce the complexity. All the best! 🙂

September 25, 2020 at 1:03 am

Thanks, Rene! Your kind words mean a lot to me! I’m so glad it has been helpful!

September 23, 2020 at 2:21 am

Honestly, I never understood stats during my entire M.Ed course and was another nightmare for me. But how easily you have explained each concept, I have understood stats way beyond my imagination. Thank you so much for helping ignorant research scholars like us. Looking forward to get hardcopy of your book. Kindly tell is it available through flipkart?

September 24, 2020 at 11:14 pm

I’m so happy to hear that my website has been helpful!

I checked on flipkart and it appears like my books are not available there. I’m never exactly sure where they’re available due to the vagaries of different distribution channels. They are available on Amazon in India.

Introduction to Statistics: An Intuitive Guide (Amazon IN) Hypothesis Testing: An Intuitive Guide (Amazon IN)

July 26, 2020 at 11:57 am

Dear Jim I am a teacher from India . I don’t have any background in statistics, and still I should tell that in a single read I can follow your explanations . I take my entire biostatistics class for botany graduates with your explanations. Thanks a lot. May I know how I can avail your books in India

July 28, 2020 at 12:31 am

Right now my books are only available as ebooks from my website. However, soon I’ll have some exciting news about other ways to obtain it. Stay tuned! I’ll announce it on my email list. If you’re not already on it, you can sign up using the form that is in the right margin of my website.

June 22, 2020 at 2:02 pm

Also can you please let me if this book covers topics like EDA and principal component analysis?

June 22, 2020 at 2:07 pm

This book doesn’t cover principal components analysis. Although, I wouldn’t really classify that as a hypothesis test. In the future, I might write a multivariate analysis book that would cover this and others. But, that’s well down the road.

My Introduction to Statistics covers EDA. That’s the largely graphical look at your data that you often do prior to hypothesis testing. The Introduction book perfectly leads right into the Hypothesis Testing book.

June 22, 2020 at 1:45 pm

Thanks for the detailed explanation. It does clear my doubts. I saw that your book related to hypothesis testing has the topics that I am studying currently. I am looking forward to purchasing it.

Regards, Take Care

June 19, 2020 at 1:03 pm

For this particular article I did not understand a couple of statements and it would great if you could help: 1)”If sample error causes the observed difference, the next time someone performs the same experiment the results might be different.” 2)”If the difference does not exist at the population level, you won’t obtain the benefits that you expect based on the sample statistics.”

I discovered your articles by chance and now I keep coming back to read & understand statistical concepts. These articles are very informative & easy to digest. Thanks for the simplifying things.

June 20, 2020 at 9:53 pm

I’m so happy to hear that you’ve found my website to be helpful!

To answer your questions, keep in mind that a central tenant of inferential statistics is that the random sample that a study drew was only one of an infinite number of possible it could’ve drawn. Each random sample produces different results. Most results will cluster around the population value assuming they used good methodology. However, random sampling error always exists and makes it so that population estimates from a sample almost never exactly equal the correct population value.

So, imagine that we’re studying a medication and comparing the treatment and control groups. Suppose that the medicine is truly not effect and that the population difference between the treatment and control group is zero (i.e., no difference.) Despite the true difference being zero, most sample estimates will show some degree of either a positive or negative effect thanks to random sampling error. So, just because a study has an observed difference does not mean that a difference exists at the population level. So, on to your questions:

1. If the observed difference is just random error, then it makes sense that if you collected another random sample, the difference could change. It could change from negative to positive, positive to negative, more extreme, less extreme, etc. However, if the difference exists at the population level, most random samples drawn from the population will reflect that difference. If the medicine has an effect, most random samples will reflect that fact and not bounce around on both sides of zero as much.

2. This is closely related to the previous answer. If there is no difference at the population level, but say you approve the medicine because of the observed effects in a sample. Even though your random sample showed an effect (which was really random error), that effect doesn’t exist. So, when you start using it on a larger scale, people won’t benefit from the medicine. That’s why it’s important to separate out what is easily explained by random error versus what is not easily explained by it.

I think reading my post about how hypothesis tests work will help clarify this process. Also, in about 24 hours (as I write this), I’ll be releasing my new ebook about Hypothesis Testing!

May 29, 2020 at 5:23 am

Hi Jim, I really enjoy your blog. Can you please link me on your blog where you discuss about Subgroup analysis and how it is done? I need to use non parametric and parametric statistical methods for my work and also do subgroup analysis in order to identify potential groups of patients that may benefit more from using a treatment than other groups.

May 29, 2020 at 2:12 pm

Hi, I don’t have a specific article about subgroup analysis. However, subgroup analysis is just the dividing up of a larger sample into subgroups and then analyzing those subgroups separately. You can use the various analyses I write about on the subgroups.

Alternatively, you can include the subgroups in regression analysis as an indicator variable and include that variable as a main effect and an interaction effect to see how the relationships vary by subgroup without needing to subdivide your data. I write about that approach in my article about comparing regression lines . This approach is my preferred approach when possible.

April 19, 2020 at 7:58 am

sir is confidence interval is a part of estimation?

April 17, 2020 at 3:36 pm

Sir can u plz briefly explain alternatives of hypothesis testing? I m unable to find the answer

April 18, 2020 at 1:22 am

Assuming you want to draw conclusions about populations by using samples (i.e., inferential statistics ), you can use confidence intervals and bootstrap methods as alternatives to the traditional hypothesis testing methods.

March 9, 2020 at 10:01 pm

Hi JIm, could you please help with activities that can best teach concepts of hypothesis testing through simulation, Also, do you have any question set that would enhance students intuition why learning hypothesis testing as a topic in introductory statistics. Thanks.

March 5, 2020 at 3:48 pm

Hi Jim, I’m studying multiple hypothesis testing & was wondering if you had any material that would be relevant. I’m more trying to understand how testing multiple samples simultaneously affects your results & more on the Bonferroni Correction

March 5, 2020 at 4:05 pm

I write about multiple comparisons (aka post hoc tests) in the ANOVA context . I don’t talk about Bonferroni Corrections specifically but I cover related types of corrections. I’m not sure if that exactly addresses what you want to know but is probably the closest I have already written. I hope it helps!

January 14, 2020 at 9:03 pm

Thank you! Have a great day/evening.

January 13, 2020 at 7:10 pm

Any help would be greatly appreciated. What is the difference between The Hypothesis Test and The Statistical Test of Hypothesis?

January 14, 2020 at 11:02 am

They sound like the same thing to me. Unless this is specialized terminology for a particular field or the author was intending something specific, I’d guess they’re one and the same.

April 1, 2019 at 10:00 am

so these are the only two forms of Hypothesis used in statistical testing?

April 1, 2019 at 10:02 am

Are you referring to the null and alternative hypothesis? If so, yes, that’s those are the standard hypotheses in a statistical hypothesis test.

April 1, 2019 at 9:57 am

year very insightful post, thanks for the write up

October 27, 2018 at 11:09 pm

hi there, am upcoming statistician, out of all blogs that i have read, i have found this one more useful as long as my problem is concerned. thanks so much

October 27, 2018 at 11:14 pm

Hi Stano, you’re very welcome! Thanks for your kind words. They mean a lot! I’m happy to hear that my posts were able to help you. I’m sure you will be a fantastic statistician. Best of luck with your studies!

October 26, 2018 at 11:39 am

Dear Jim, thank you very much for your explanations! I have a question. Can I use t-test to compare two samples in case each of them have right bias?

October 26, 2018 at 12:00 pm

Hi Tetyana,

You’re very welcome!

The term “right bias” is not a standard term. Do you by chance mean right skewed distributions? In other words, if you plot the distribution for each group on a histogram they have longer right tails? These are not the symmetrical bell-shape curves of the normal distribution.

If that’s the case, yes you can as long as you exceed a specific sample size within each group. I include a table that contains these sample size requirements in my post about nonparametric vs parametric analyses .

Bias in statistics refers to cases where an estimate of a value is systematically higher or lower than the true value. If this is the case, you might be able to use t-tests, but you’d need to be sure to understand the nature of the bias so you would understand what the results are really indicating.

I hope this helps!

April 2, 2018 at 7:28 am

Simple and upto the point 👍 Thank you so much.

April 2, 2018 at 11:11 am

Hi Kalpana, thanks! And I’m glad it was helpful!

March 26, 2018 at 8:41 am

Am I correct if I say: Alpha – Probability of wrongly rejection of null hypothesis P-value – Probability of wrongly acceptance of null hypothesis

March 28, 2018 at 3:14 pm

You’re correct about alpha. Alpha is the probability of rejecting the null hypothesis when the null is true.

Unfortunately, your definition of the p-value is a bit off. The p-value has a fairly convoluted definition. It is the probability of obtaining the effect observed in a sample, or more extreme, if the null hypothesis is true. The p-value does NOT indicate the probability that either the null or alternative is true or false. Although, those are very common misinterpretations. To learn more, read my post about how to interpret p-values correctly .

March 2, 2018 at 6:10 pm

I recently started reading your blog and it is very helpful to understand each concept of statistical tests in easy way with some good examples. Also, I recommend to other people go through all these blogs which you posted. Specially for those people who have not statistical background and they are facing to many problems while studying statistical analysis.

Thank you for your such good blogs.

March 3, 2018 at 10:12 pm

Hi Amit, I’m so glad that my blog posts have been helpful for you! It means a lot to me that you took the time to write such a nice comment! Also, thanks for recommending by blog to others! I try really hard to write posts about statistics that are easy to understand.

January 17, 2018 at 7:03 am

I recently started reading your blog and I find it very interesting. I am learning statistics by my own, and I generally do many google search to understand the concepts. So this blog is quite helpful for me, as it have most of the content which I am looking for.

January 17, 2018 at 3:56 pm

Hi Shashank, thank you! And, I’m very glad to hear that my blog is helpful!

January 2, 2018 at 2:28 pm

thank u very much sir.

January 2, 2018 at 2:36 pm

You’re very welcome, Hiral!

November 21, 2017 at 12:43 pm

Thank u so much sir….your posts always helps me to be a #statistician

November 21, 2017 at 2:40 pm

Hi Sachin, you’re very welcome! I’m happy that you find my posts to be helpful!

November 19, 2017 at 8:22 pm

great post as usual, but it would be nice to see an example.

November 19, 2017 at 8:27 pm

Thank you! At the end of this post, I have links to four other posts that show examples of hypothesis tests in action. You’ll find what you’re looking for in those posts!

Comments and Questions Cancel reply

Hypothesis Testing: Does Chance explain the Results?

This chapter discusses rules for deciding between competing hypotheses on the basis of data that have a random component (such as draws from a box of tickets). The competing hypotheses are called the null hypothesis and the alternative hypothesis . The rules are called hypothesis tests or hypothesis testing procedures . Typically, the null hypothesis is that something is not present, that a treatment has no effect, or that there is no difference between two parameters . Typically, the alternative hypothesis is that some effect is present, that a treatment has an effect, or that two parameters differ. The main requirement of the null hypothesis is that it must be possible to compute the probability that the test rejects the null hypothesis when the null hypothesis is true. That probability is called the significance level of the test. (When in doubt, choose the simpler of the hypotheses to be the null hypothesis—usually that will lead to easier computations.)

The two types of error are as follows:

• Rejecting a true null hypothesis. This is called a Type I error . In ordinary language, a Type I error is a false alarm.
• Failing to reject a false null hypothesis. This is called a Type II error .

Controlling the chances of these two kinds of error is crucial.

Examples of Hypothesis Testing Problems

Many questions we encounter daily can be cast as hypothesis testing problems. Here are some examples:

• Airport security systems are designed to detect weapons and bombs. When you walk through an airport metal detector, the system is trying to discriminate between the hypothesis "this person is not carrying a weapon" and the hypothesis "this person is carrying a weapon," on the basis of electromagnetic field measurements. The measurements are subject to various uncertainties, and other objects you might be carrying can perturb the field in ways that are hard to distinguish from the perturbations produced by weapons. As a result, the system cannot determine with perfect accuracy whether you are carrying a weapon. The security system can make two kinds of error: setting off the alarm when you do not have a weapon, and failing to set off the alarm when you are carrying a weapon.
• At a dental check-up, the dentist tries to discriminate between the hypothesis that your teeth are fine, and the hypothesis that you have one or more cavities, on the basis of data collected by looking in your mouth, poking and prodding, and perhaps taking X-rays. These measurements are subject to uncertainties. As a result, the dentist can make two kinds of errors: concluding you have a cavity when you do not (usually this is cleared up by X-rays), or failing to notice a cavity that you do have. The null hypothesis is that you do not have a cavity; the alternative hypothesis is that you have one or more cavities. A Type I error occurs if the dentist concludes you have a cavity, but you do not. A Type II error occurs if the dentist fails to notice a cavity.
• A drug company wants to determine whether a new headache remedy works. The conclusion will be based on what happens when a group of subjects use the remedy or a placebo to treat headaches. Most headaches eventually go away on their own, and some headaches (or some peoples' headaches) are hard to relieve, so the company can make two kinds of mistake: incorrectly concluding that the remedy works when in fact it does not, and failing to notice that an effective remedy works. The null hypothesis is that the remedy does not relieve headaches; the alternative hypothesis is that it does. A Type I error occurs if the company concludes that the remedy works when in fact it does not. A Type II error occurs if the remedy is effective, but the company concludes that it is not.
• A seismologist claims to be able to predict earthquakes. She issues some predictions; some of them come true. How might we decide whether the method has merit? There are two kinds of errors we could make: erroneously concluding that the method has merit when it does not, and failing to recognize that the method has merit when it does. The null hypothesis is that the prediction method does not work—that its successes are coincidental. The alternative hypothesis is that the method works. A Type I error occurs if we conclude that the prediction method has merit when it does not. A Type II error occurs if we fail to conclude that the prediction method has merit when it works.
• A user runs a query on a search engine to look for information on the web. For each document in its index, the search engine tries to decide between the hypothesis "this document is not interesting to the user" and the hypothesis "this document is interesting to the user." There are two kinds of errors the search engine can make: showing the user a document the user is not interested in, and failing to show the user a document that the user would have found interesting. The null hypothesis is that the document is not interesting to the user; the alternative hypothesis is that the document is interesting to the user. A Type I error occurs if the engine shows the user a document that is not interesting. A Type II error occurs if the engine fails to show the user a document the user would find interesting. (The fraction of retrieved documents that are in fact interesting is called the precision . The fraction of all interesting documents the search engine in fact retrieves is called the recall . Precision and recall are related to the rates of Type I and Type II errors.)
• Internet content filters try to block users from seeing web pages deemed objectionable or harmful according to some standard. When a user requests a given page, the content filter tries to decide between the hypothesis "this page should not be blocked" and the hypothesis "this page should be blocked." There are two kinds of errors the filter can make: showing someone a page that should be blocked according to the standard, and blocking a page that should not be blocked according to the standard. The null hypothesis is that the page should not be blocked; the alternative is that the page should be blocked. A Type I error occurs if the filter blocks a page that should not be blocked. (This is called overblocking .) A Type II error occurs if the filter fails to block a page that should be blocked. (This is called underblocking .)
• Post-election audits try to determine whether the apparent outcome of an election is the same outcome that a complete manual count of the paper audit trail would show. Post-election audits hand count the votes in a random sample of precincts and compare those counts with the original machine counts. The goal of a risk-limiting audit is to ensure that the chance of a full manual count is high whenever the full manual count would change the winner. It is virtually inevitable that a hand count will find at least some difference in the number of votes for each candidate. There are two kinds of mistake that the audit can make: concluding that the right person won when in fact he or she did not, and requiring a full manual count when in fact that will show the same answer as the machine count. The null hypothesis is that a full manual count would show a different winner than the machine count did; the alternative is that the hand and machine counts would show the same winner. A Type I error occurs if the audit does not require a full manual count when that would show a different winner—if the audit does not detect a problem that is really there. A Type II error occurs if the audit requires a full manual count that results in the same winner the machine count shows—if the audit requires an unnecessary full manual count.

How to Tell the Liars from the Statisticians (R. Hooke, 1983. Marcel Dekker, Inc., NY, 173pp) characterizes the difference between liberal and conservative politics in terms of Type I and Type II errors. In offering public assistance, such as welfare, a Type I error is to give public assistance to someone who is not really deserving, and a Type II error is to fail to give support to someone who really needs it. In this context, conservatives tend to find Type I errors intolerable, and liberals tend to find Type II errors intolerable. In punishing crime, the opposite is true: our legal system holds that someone is innocent until proven guilty, so a Type I error occurs if an innocent person is punished, and a Type II error occurs if a guilty person is not punished. Here, liberals tend to find Type I errors intolerable, and conservatives tend to find Type II errors intolerable. Because it is not possible to eliminate one type of error without increasing the frequency of the other type of error, the two political philosophies are at odds because they advocate opposite extremes of the error tradeoff.

The following exercises check your ability to identify null and alternative hypotheses, and Type I and Type II errors.

Significance Level and Power

The power of an hypothesis test against a specific alternative hypothesis is the chance that the test correctly rejects the null hypothesis when that alternative hypothesis is true; that is, the power is 100% minus the chance of a Type II error when that alternative hypothesis is true. The chance of a Type II error is often denoted by the lowercase Greek letter beta ( β ), so the power is (100% − β) .

The significance level and the power of a test are the probability of the same event, the event that the null hypothesis is rejected. The difference between significance level and power is the assumption about the world we use to compute the probability: to compute the significance level , we assume that the null hypothesis is true; to compute the power , we assume that the alternative hypothesis is true.

The significance level of an hypothesis test is the chance that the test rejects the null hypothesis , on the assumption that the null hypothesis is true.

The power of an hypothesis test against a particular alternative hypothesis is the chance that the test rejects the null hypothesis , on the assumption that that alternative hypothesis is true.

If the gun is not loaded, it will not go off either time the trigger is pulled, so the test cannot reject the null hypothesis erroneously: the significance level of the test is zero.

Suppose one chamber is loaded. The chance that the gun goes off is 100% minus the chance that it does not go off. It does not go off if the loaded chamber is one of the four that are not tried. The chance of that is 4/6, so the chance that the gun does go off if one chamber is loaded is 2/6 = 1/3. Thus the power of the test against the alternative that one chamber is loaded is 1/3.

If five chambers are loaded, there is only one empty chamber, so if the trigger is pulled twice, the gun will go off at least once. Thus the power of the test against the alternative that five chambers are loaded is 100%.

Test Statistics and P -values

P( X ≥ x ) ≤ p.

Then for any p between 0 and 100%, the rule

reject the null hypothesis if X ≥ x p

tests the null hypothesis at significance level p . If we observed X = x , the P -value of the null hypothesis given the data would be the smallest p such that x ≥ x p .

The smaller the P -value , the stronger the evidence against the null hypothesis .

The basic steps in statistical hypothesis testing are:

• Formulate the null and alternative hypotheses.
• Specify the maximum permissible chance of a Type I error (the significance level of the test).
• Choose the procedure that will be used to test the hypothesis. Typically, the procedure is to compute a test statistic from the data, and to reject the null hypothesis if the value of the test statistic is in some rejection region . The rejection region is determined by insisting that the significance level be the value chosen in the previous step: the chance that the statistic is in the rejection region if the null hypothesis be true must be no larger than the significance level .
• Collect the data.
• Compute the test statistic. Reject the null hypothesis if the test statistic falls in the rejection region; otherwise, do not reject the null hypothesis. (Alternatively, report the P -value of the null hypothesis .)

Zener cards often are used to test claims of extra-sensory perception (ESP), such as telepathy (mind reading) and clairvoyance (knowing something without perceiving it with the usual five senses). Each Zener card has one of five geometric figures on it: a star, a square, a circle, wavy lines, or a plus sign. Zener cards were developed by Dr. Karl Zener, of Duke University, and were first used to study ESP by Dr. J.B. Rhine (1895–1980), who allegedly coined the term extra-sensory perception .

Consider using Zener cards to test a psychic's claimed ability to sense what card someone is looking at. Imagine shuffling the five cards well, then looking at each one in turn (without showing the card to the psychic). Each time we look at a card, the psychic writes down which of the five cards she thinks we are looking at.

• We do not look at what the psychic wrote until we have gone through the whole deck.
• The psychic must assign every symbol to exactly one card. For example, the psychic cannot say that the first card and the third card both are labeled with circles.

We do not reveal the correct answer to the psychic, nor tell her whether her determination was right or wrong, until we have passed through the entire deck. Otherwise, she could use that information to improve subsequent guesses. For example, if the psychic gets to see each card after making her determination, she is guaranteed to be able to get the last card right by a process of elimination. We should not learn the psychic's determinations until the test is over, so that we do not inadvertantly give information away through facial expressions, etc . We insist that the psychic not repeat a symbol; otherwise, the psychic could be certain to identify at least one card correctly, merely by repeating a single determination five times (for example, saying every card is marked with the circle).

The following exercises test your understanding of the Zener card example.

Videos of Exercises

(Reminder: Examples and exercises may vary when the page is reloaded; the video shows only one version.)

The following exercises check your ability to use this probability distribution to construct an hypothesis test for ESP.

Hypotheses about parameters: One-sided and Two-sided Alternative Hypotheses.

Quite commonly, the null hypothesis is that a parameter μ equals some particular value a (the null value ), and the alternative hypothesis is that μ is greater than a , that μ is less than a , or simply that μ is not equal to a . (The first two are one-sided alternative hypotheses; the last is a two-sided alternative hypothesis.) Many of the examples from the beginning of this chapter can be written this way.

• In the airport security example, the null hypothesis could be written (number of weapons = 0). The alternative hypothesis could be written (number of weapons > 0).
• In the dental examination, the null hypothesis could be written (number of cavities = 0). The alternative hypothesis could be written (number of cavities > 0).
• In the headache remedy example, the null hypothesis could be written (effect on headache duration = 0). The alternative hypothesis could be written (effect on headache duration < 0) [the remedy decreases the duration of headaches].
• In the manufacturing example, the null hypothesis could be written (old defect rate − new defect rate = 0). The alternative hypothesis could be written (old defect rate − new defect rate > 0).

In these examples, all the alternative hypotheses are one-sided : they assert that the value of the parameter μ is on one side of the null value a . That is, each null hypothesis asserts that μ = a , and each alternative hypothesis either asserts that μ < a , or it asserts that μ > a . In contrast, if we wanted to test whether a coin was fair, the null hypothesis would be (chance of tails = 50%), and the alternative hypothesis could be (chance of tails >50% or <50%). That is a two-sided alternative hypothesis: it asserts that μ is not equal to a .

A good test has as much power as it can against every plausible alternative—while maintaining its significance level. An hypothesis test about the value of a parameter that is designed to have as much power as possible against alternative values of the parameter on both sides of the null value is called a two-sided test . A test that is designed to have as much power as possible against alternative values of the parameter on only one side of the null value is called a one-sided test .

To pick a rejection region given a test statistic X , a null hypothesis, and an alternative hypothesis, we think about how the distribution of the test statistic under the null hypothesis differs from its distribution under the alternative hypothesis. If the test statistic is likely to be larger if the alternative hypothesis be true than if the null hypothesis be true, it makes sense to use a rejection region of the form {X > x 0 } ; we would choose x 0 so that if the null hypothesis is true, the chance that X > x 0 is at most the significance level. If the test statistic is likely to be smaller if the alternative hypothesis is true than if the null hypothesis is true, it makes sense to use a rejection region of the form {X < x 0 } ; we would choose x 0 so that if the null hypothesis is true, the chance that X < x 0 is at most the significance level. If the test statistic is likely to be further from some reference point x 0 if the alternative hypothesis be true than if the null hypothesis be true, it makes sense to use a rejection region of the form

{X < x 1 ∪   X > x 2 }.

we would choose x 1 and x 2 so that the chance that X is in the rejection region if the null hypothesis is true is at most the significance level; we would also tend to choose them so that the probability that X < x 1 is equal to the probability that X > x 2 if the null hypothesis is true. The following exercises check whether you understand when to use a one-sided test and when to use a two-sided test.

Case Study: Employment Discrimination Arbitration

This example is based on a true story. The names have been changed, but other than that, the facts are stated as I understand them.

Service, Inc., provides janitorial services under contract to large organizations. Because of the nature of their business, the turnover of their service employees tends to be somewhat high. A number of people who had been fired from service positions at a particular branch of Service, Inc., between 22 June 1996 and 8 September, 1997, filed suit against Service, Inc., claiming that they were discriminated against on the basis of gender, age, and/or ethnicity. In particular, the suit alleged that women over the age of 40 were fired more often than other groups. I was retained in late 1997 to examine summary employment data for evidence of discrimination on the basis of gender, age, and ethnicity.

I was given summary employment listings for 143 service employees who had worked for Service, Inc., at that location, at any time between 22 June 1996 and 8 September 1997. The summary listings included age, gender, and ethnicity, for all but two of the employees. Those employees had Hispanic surnames; I imputed their ethnicity to be Hispanic. The summary listings also indicated whether the employee was still working for Service, and if not, whether their termination was voluntary (resignation, leave-of-absence, etc.) or involuntary (the person was fired). Among the 143 entries, 24 recorded involuntary terminations; the remaining 119 were for individuals who were still employed, on leave, or had left voluntarily.

I divided the employees into two groups by age: those whose employment by Service, Inc., ended before their 40th birthday, or who were still employed but were not yet 40 years old as of 8 September 1997; and those whose employment ended after their 40th birthday, or who were still employed but were at least 40 years old as of 8 September 1997.

*Ethnicity: 1, White; 2, Black; 3, Hispanic; 4, Native American/Alaskan; 5, Asian and Pacific Islander.

How might we assess whether Service, Inc., discriminated in firing on the basis of age, gender, and/or ethnicity? One way is to ask whether the age, gender, and ethnicity breakdown of the 24 involuntarily terminated employees is surprisingly different from the breakdown that would be expected had 24 of the 143 employees been selected at random. This is not to suggest that people really are fired at random, nor that competence, reliability, and adequate job performance are necessarily equal (even on the average) for different demographic groups. Rather, the question is whether the assumption that involuntary terminations were blind to age, gender, and ethnicity is compatible with the data. We shall take the total number of people terminated involuntarily as a given, 24 (we shall condition on the number of people terminated involuntarily).

For example, consider the table of termination by gender . Of the 143 employees, 81 were female (81/143 = 56.64%); of the 24 employees who were fired, 14 were female (14/24 = 58.33%). Suppose that 24 employees were selected at random without replacement from the 143 employees in the period in question. Would it be surprising if 14 or more of those 24 employees were women?

There is a Federal case, Equal Employment Opportunity Commission v. Federal Reserve Bank of Richmond , 673 F.2d 798 (1983), that says one should look at whether the firing rates are surprisingly large or surprisingly small in making this assessment (in statistical parlance, one should use two-sided rather than one-sided hypothesis tests). That is, we should look at the mean number of women in all possible simple random samples of 24 employees from the 143, and look at the difference between that average and the number of women actually fired. If that difference is surprisingly large (if the number fired is much larger or much smaller than the average), there is prima facie evidence of discrimination—possibly reverse discrimination. If differences that large or larger are relatively likely to occur in a simple random sample of 24 employees from the 143, there is no prima facie evidence of discrimination: the "luck of the draw" is sufficient to explain the observed difference.

The number of women in a simple random sample from the employees is like the number of tickets labeled "1" in n draws without replacement from a box of N tickets of which G are labeled "1" and the rest are labeled "0" is n×G/N . That number has an hypergeometric distribution with parameters N , G and n . We saw previously that the expected value of the number of tickets labeled "1" is n×G/N . The expected value of the number of women in a simple random sample of 24 employees is thus

24 × (81/143) = 13.594 women.

One would not expect to see a fractional number of women in the sample; nonetheless, this is how the expected value is defined (it is the long-run average number of women in repeated simple random samples of size 24, or the probability-weighted average of the possible number of women in the sample). Because of the luck of the draw, the number of women in the sample will vary from draw to draw, but it is likely to be in a range around 14. The chance of each possible number of women in the sample (from 0 to 24) is given by the hypergeometric distribution .

Similarly, the chance of each possible number of people under the age of 40 in the sample, and of the number of people of each ethnicity in the sample, all have hypergeometric distributions with N = 143 and n = 24 , but with difference values of G .

What happens when we want to look at more than two groups at a time, for example, the breakdown by age and gender? For example, what is the chance that a random sample of 24 of the employees has 7 women under 40, 7 women 40 or older, 7 men under 40, and 3 men 40 or over?

If we were to make a box model for the draws, the tickets in the box would have more than two labels (male under 40, male 40+, female under 40, female 40+), so this is not like the number of tickets labeled "1" in draws from a box that has tickets labeled "0" and "1." However, we can use the same kind of reasoning we have used in the last few chapters to figure out the chance. It is just like the chance of a card hand, dealing a 24-card hand from a deck of 143 cards that has only one suit, and four kinds of cards, with different numbers of each kind of cards. One kind of card corresponds to males under 40, one kind to males 40 and over, etc.

The total number of ways to draw 24 employees without replacement from the 143 is 143 C 24 . These are equally likely in simple random sampling —indeed, that is the definition of a simple random sample. How many of those ways result in 7 women under age 40, 7 women age 40 or older, 7 men under age 40, and 3 men age 40 or over? We can use the fundamental rule of counting . There are 32 female employees under age 40, so there are 32 C 7 ways to select 7 of them. There are 49 female employees age 40 and over, so there are 49 C 7 ways to select 7 of them. There are 43 male employees under age 40, so there are 43 C 7 ways to select 7 of them. There are 19 male employees age 40 and over, so there are 19 C 3 ways to select 3 of them. By the fundamental rule of counting , there are

32 C 7 × 49 C 7 × 43 C 7 × 19 C 3

ways to make all these choices. The chance that a simple random sample of 24 employees has 7 women under age 40, 7 women age 40 or older, 7 men under age 40, and 3 men age 40 or over is thus

To assess whether the data evidence discrimination, I calculated the chance that the ethnicity, gender, and age proportions of a group of 24 employees chosen at random from the population of 143 would differ from the corresponding proportions among the 143 by as much or more than observed. These are the P -values the null hypotheses of no discrimination on the basis of

• Gender or age
• White vs. non-white
• Gender or ethnicity

Conclusions

If 24 of the 143 employees were terminated completely at random, there would be more than an 80% chance that the proportions of protected minorities terminated involuntarily would differ from their corresponding proportions in the employee pool by at least as much as these data show. The data are quite consistent with the hypothesis that there was no discrimination by age, ethnicity, or gender. Empirically, employees age 40 and over were less likely to be fired than employees under age 40, with women age 40 and over less likely to be fired than men age 40 and over.

The following exercise asks you to test the null hypothesis that two probabilities are equal. As is the case in the previous example of no discrimination, the null hypothesis—that a decision is made randomly—is contrived. Nonetheless, the large P -value is suggestive.

Hypothesis tests need to be interpreted with care. Rejecting the null hypothesis does not mean that the null hypothesis is false, nor does failing to reject the null mean that the null hypothesis is true. Practical importance and statistical significance have little to do with each other. P -values often are misinterpreted. The number of tests performed matters. The fraction of rejected null hypotheses that are rejected in error depends on more than just the significance level.

The Meaning of Rejection

In testing hypotheses, we speak of rejecting the null hypothesis or not rejecting the null hypothesis. We do not speak of accepting the null hypothesis or the alternative hypothesis . Statisticians use data to show that some possibilities are implausible, but there are always many possible explanations for the data we observe. Moreover, if the data are poor or few in number, typically they cannot provide strong evidence against the null hypothesis, even if the null hypothesis be false. We should not interpret poor or inconclusive data as supporting the null hypothesis. Sometimes we can rule out an hypothesis as being inconsistent with the data (if the data are extremely unlikely on the assumption that the hypothesis are true), but the set of hypotheses that are consistent with the data usually contains more than just the null and alternative hypotheses. The precise statistical statement when we reject a null hypothesis is that either the null hypothesis is false, or an event has occurred that has probability no larger than the significance level.

Rejecting the null hypothesis

Not rejecting the null hypothesis does not mean that the null hypothesis is true, nor that the data support the null hypothesis.

In particular, if the data are few or poor, it is hard for a test to have much power—it is hard for a test to reject a false null hypothesis.

Rejecting the null hypothesis does not mean that the alternative hypothesis is true.

It means that either the null hypothesis is false, or an event has occurred that has probability no larger than the significance level.

It is not hard to a construct "straw man" null hypothesis that will succumb to the slightest contact with data.

Statistical Significance and Practical Importance

If the null hypothesis is rejected, one says that the effect or test is "statistically significant at level___," where the significance level or the P -value goes in the blank. "At level___" often is omitted, which makes it impossible to know what the chance of a false alarm might be. All too often, the word "statistically" is dropped too, leading one to think that the effect is important, not merely detectable. The difference between importance and detectability is considerable. A small, unimportant effect can be detected if there are sufficiently many data of sufficiently high quality. Conversely, an effect can be both large and important, but not statistically significant if the data are few or of low quality. That can lead to peculiar locutions, such as "no other leading brand has been shown to surpass ZZZ." Aside from the ambiguity in the word "leading," one might not reject the null hypothesis that no brand is better than ZZZ because ZZZ really is at least as good as all other brands, or because the data are too few or of too low quality to allow one to detect that another brand actually is better than ZZZ.

Practical significance (importance) and statistical significance (detectability) have little to do with each other.

An effect can be important, but undetectable (statistically insignificant) because the data are few, irrelevant, or of poor quality.

An effect can be statistically significant (detectable) even if it is small and unimportant, if the data are many and of high quality.

Interpreting P -values

A common mistake in hypothesis testing is to misinterpret the P -value or significance level; in particular, to consider the P -value or significance level to be the probability that the null hypothesis is true. The data have a random component, but the truth of the null hypothesis is not random—the null hypothesis is either true or false, regardless of what data we observe. The P -value is a probability computed on the assumption that the null hypothesis is true. As is the case for confidence intervals , chance is meaningful only before the data are collected. The null hypothesis is either true, or not. Once the data have been collected, there is no chance left: the hypothesis testing procedure either rejects the null hypothesis, or not. Depending on whether the null hypothesis is true, an error occurs, or not.

Multiplicity and Data Mining

The significance levels we have been computing are for testing a single hypothesis with a single test. Suppose we were interested in whether any of the "brain cocktails" sometimes served at parties is effective in increasing mental acuity.

We test the effectiveness of 10 different types of cocktails using the methodology described in this chapter, using a 5% significance level for each test. We use different individuals to test each kind of cocktail; we assume that the outcomes of the tests are independent. This is an example of multiplicity : testing more than one hypothesis simultaneously.

Suppose we go through the protocol, and cocktail X shows up as having a significant effect; that is, we reject the hypothesis that cocktail X has no effect, at significance level 5%. On the face of it, it appears that cocktail X improves mental acuity (the cocktail increases acuity, however we measure it, or an event has occurred that has chance no larger than 5%). It would seem, therefore, that we could reject the null hypothesis that none of the brain cocktails is effective, at significance level 5%—but that is not the correct significance level. The question we need to ask is, "if none of the cocktails really had an effect, what would be the chance of getting at least one positive result?"

The grand null hypothesis is that none of the cocktails makes a difference. The alternative hypothesis is that at least one of them improves mental acuity. If the grand null hypothesis is true, what is the chance that at least one of the tests gives a false positive result?

If the grand null hypothesis is true, the number of false positives would have the same distribution as the number of 1's in 10 draws with replacement from a 0-1 box that has 5 tickets labeled "1" and 95 tickets labeled "0," and adding the draws.

That distribution is binomial, with parameters n = 10 and p = 5% . The chance that we get at least one ticket labeled "1" is 100% minus the chance that we get no ticket labeled "1":

chance of at least one false positive = 100% − (chance of no false positive)

= 100% − (95%) 10

= 100% − 60%

Even though we test the individual hypotheses that each cocktail has no effect at significance level 5%, the resulting significance level for testing the "grand" null hypothesis is 40%.

This is typical of the effect of multiplicity on the significance level: The more tests performed, the greater the chance of a false positive. If you test many hypotheses at, say, 5% significance level, using independent data, you should expect that in the long run you would erroneously reject about 5% of the null hypotheses that are in fact true.

Beware studies that apply many different tests or test many different hypotheses from the same data, and claim a significant result. Often, such studies neglect the effect of multiplicity, and the chance of a false positive is much higher than the authors recognize. Applying many hypothesis tests to the same data in search of a significant result is known in Statistics as "data mining" or "data snooping."

Garbage in, garbage out

Often in science it is the hypothesis rejections that are interesting. Typically, rejecting a null hypothesis means deciding that some effect is present or important; this is called a "discovery." At one extreme, every discovery is true—every rejected null hypothesis is in fact false. At the other extreme, every discovery is false—every rejected null hypothesis is in fact true. Typically, only the "discoveries" are brought to our attention. Few scientists seek to publish negative results. Consequently, we see primarily the rejections in tests of a population of hypotheses of which an unknown fraction really are false. Testing hypotheses at significance level 5% does not mean that 5% of the rejections are erroneous. The fraction of erroneous rejections depends on the fraction of true null hypotheses, and can be anywhere between 0% and 100%, regardless of the significance level of the tests.

Suppose one tests a large collection of hypotheses. Among those that are not rejected, what fraction are true? Among those that are rejected, what fraction are false? This question cannot be answered unless one knows what proportion of null hypotheses tested are false, as simple reasoning shows:

Suppose a fraction t of the null hypotheses tested are in fact true, so the fraction of null hypotheses that are false is (1−t) . If t = 100% , every hypothesis that is rejected is rejected erroneously, and every hypothesis that is not rejected is really true—every error we make is a Type I error. At the other extreme, if t = 0 , every hypothesis that is rejected is rejected correctly, and every hypothesis that is not rejected is in fact false—every error we make is a Type II error. No matter how good the test is, it cannot make a true hypothesis false, nor a false hypothesis true. Unless the test always rejects, if fed a steady diet of false hypotheses, it will fail to reject some of them. Unless the test never rejects, if fed a steady diet of true hypotheses, it will erroneously reject some of them. These remarks can be summarized as "garbage in, garbage out."

Suppose that every test performed has the same power. The chance that a false null hypothesis is rejected is the power, and the chance that a true null hypothesis is rejected is the significance level, so the long-run fraction of rejected hypotheses that are in fact false is

(the numerator is the expected fraction of false hypotheses that are rejected; the denominator is the sum of the expected fraction of false hypotheses that are rejected and the expected fraction of true hypotheses that are rejected). The long-run fraction of hypotheses that are not rejected and that are in fact true is

(the numerator is the expected fraction of true hypotheses that are not rejected; the denominator is the sum of the expected fraction of true hypotheses that are not rejected and the expected fraction of false hypotheses that are not rejected).

Relatively recently, statisticians have developed hypothesis testing methods that keep the rate of false discoveries under control (see the work of Benjamini, Hochberg, and others). That is, the methods guarantee that the fraction of "discoveries" that are erroneous rejections of true null hypotheses does not exceed a specified limit (such as 5%). These methods control what is called "the false discovery rate." Perhaps those methods will be used commonly in the future. Until then, it is prudent to keep in mind that the proportion of real discoveries among the claims of "discoveries" is unknown. Scientists are much more likely to report positive results than negative ones, and scientific journals are much more likely to publish positive reports than negative ones, so the majority of hypothesis tests that are reported are the "discoveries." The failures to reject null hypotheses are rarely reported. Thus it is plausible that many published "discoveries" are in fact erroneous rejections of null hypotheses.

Many scientific and practical questions can be posed as decisions between competing hypotheses or theories about the world: a null hypothesis and an alternative hypothesis . Two kinds of error are possible: rejecting a true null hypothesis (a Type I error ), and failing to reject a false null hypothesis (a Type II error ). If the data on which the decision is based have a random component, the rule is called a statistical hypothesis test or a test of significance . The chance an hypothesis test commits a Type I error is the significance level of the test—the chance of a false alarm. The chance that the test correctly rejects the null hypothesis when a particular alternative hypothesis is true is the power of the test against that alternative. The chance of a Type II error when a particular alternative is true is 100% minus the power against that alternative. When an hypothesis test rejects the null hypothesis, either the null hypothesis is false, or an event occurred that has probability no larger than the significance level of the test. It does not mean that the alternative hypothesis is true. When the null hypothesis is not rejected, it does not mean that the null hypothesis is true. In particular, if the data are few, irrelevant, or poor, any test will have trouble rejecting a false null hypothesis. Given a family of hypothesis tests that allow a null hypothesis to be tested at any significance level between 0 and 100%, the P -value of the null hypothesis for the observed data is the smallest significance level for which any of the tests would reject the null hypothesis.

Statistical significance should not be confused with practical importance. Statistical significance has to do with detectability, which depends on the number and quality of data, among other things. The significance level or P -value is not the probabiity that the null hypothesis is true. In fact, the P -value and significance level are both computed using the assumption that the null hypothesis is true. The power is computed using the assumption that the alternative hypothesis is true.

Hypothesis tests usually are based on a test statistic : a random variable computed from the data. If the test statistic is in the rejection region , the null hypothesis is rejected. The rejection region is chosen subject to the constraint that if the null hypothesis is true, the chance that the test statistic will be in the rejection region is at most the significance level, so that the test has the desired significance level. The rejection region also should also be chosen so that the test will have good power against the alternatives that are contemplated: The chance that the test statistic is in the rejection region should be higher when the alternative hypothesis is true than when the null hypothesis is true. For significance levels to be meaningful, the null hypothesis, test statistic, significance level, and rejection region all must be chosen before the data are collected.

Many hypotheses can be written in terms of the value of a parameter μ , the null hypothesis being that μ = μ 0 , the null value ; and the alternative hypothesis being that μ≠μ 0 (a two-sided alternative hypothesis), that μ>μ 0 (a one-sided alternative hypothesis), or that μ<μ 0 (a one-sided alternative hypothesis). Hypothesis tests designed to have good power against two-sided alternatives are called two-sided tests; tests designed to have good power only against one-sided alternatives are called one-sided tests.

The significance level of a test controls the long-run fraction of true null hypotheses that are rejected erroneously, but not the fraction of rejected null hypotheses that are rejected erroneously: That fraction depends on the significance level, the power, and on the fraction of null hypotheses that are true. If every tested null null hypothesies tested is false, every rejection is a correct rejection; if no tested null hypothesis tested is false, every rejection is an erroneous rejection, no matter how good the test. When many hypotheses are tested, the there is a the chance of that at least one Type I error will occur is much larger than the chance of a Type I error in each test; this is the issue of multiplicity . Because only rejections of null hypotheses tend to be reported in scientific literature (as "discoveries"), it is likely that a noticeable fraction of scientific results are Type I errors—false discoveries.

• alternative hypothesis
• binomial distribution
• chance variability
• confidence interval
• expected value
• false discovery rate
• Fundamental Rule of counting
• hypergeometric distribution
• hypothesis test
• independent
• mean average
• null hypothesis
• permutation
• prima facie
• probability
• probability distribution
• random sample
• random variable
• rejection region
• significance level
• simple random sample
• test statistic
• Type I error
• Type II error

• School Guide
• Mathematics
• Number System and Arithmetic
• Trigonometry
• Probability
• Mensuration
• Maths Formulas
• Class 8 Maths Notes
• Class 9 Maths Notes
• Class 10 Maths Notes
• Class 11 Maths Notes
• Class 12 Maths Notes

Tests of Significance: Process, Example and Type

Test of significance is a process for comparing observed data with a claim(also called a hypothesis), the truth of which is being assessed in further analysis. Let’s learn about test of significance, null hypothesis and Significance testing below.

Tests of Significance in Statistics

In technical terms, it is a probability measurement of a certain statistical test or research in the theory making in a way that the outcome must have occurred by chance instead of the test or experiment being right. The ultimate goal of descriptive statistical research is the revelation of the truth In doing so, the researcher has to make sure that the sample is of good quality, the error is minimal, and the measures are precise. These things are to be completed through several stages. The researcher will need to know whether the experimental outcomes are from a proper study process or just due to chance.

The sample size is the one that primarily specifies the probability that the event could occur without the effect of really performed research. It may be weak or strong depending on a certain statistical significance. Its bearings are put into question. They may or may not make a difference. The presence of a careless researcher can be a start of when a researcher instead of carefully making use of language in the report of his experiment, the significance of the study might be misinterpreted.

Significance Testing

Statistics involves the issue of assessing whether a result obtained from an experiment is important enough or not. In the field of quantitative significance, there are defined tests that may have relevant uses. The designation of tests depends on the type of tests or the tests of significance are more known as the simple significance tests.

These stand up for certain levels of error mislead. Sometimes the trial designer is called upon to predefine the probability of sampling error in the initial stage of the experiment. The population sampling test is regarded as one which does not study the whole, and as such the sampling error always exists. The testing of the significance is an equally important part of the statistical research.

Null Hypothesis

Every test for significance starts with a null hypothesis H 0 . H 0 represents a theory that has been suggested, either because it’s believed to be true or because it’s to be used as a basis for argument, but has not been proved. For example, during a clinical test of a replacement drug, the null hypothesis could be that the new drug is not any better, on average than the present drug. We would write H 0 : there’s no difference between the 2 drugs on average.

Process of Significance Testing

In the process of testing for statistical significance, the following steps must be taken:

Step 1: Start by coming up with a research idea or question for your thesis. Step 2: Create a neutral comparison to test against your hypothesis. Step 3: Decide on the level of certainty you need for your results, which affects the type of sign language translators and communication methods you’ll use. Step 4: Choose the appropriate statistical test to analyze your data accurately. Step 5: Understand and explain what your results mean in the context of your research question.

Types of Errors

There are basically two types of errors:

Type I Error

Type ii error.

Now let’s learn about these errors in detail.

A type I error is where the researcher finds out that the relationship presumed maxim is a case; however, there is evidence showing it is not a function explained. This type of error leads to a failure of the researcher who says that the H 0 or null hypothesis has to be accepted while in reality, it was supposed to be rejected together with the research hypothesis. Researchers commit an error in the first type when α (alpha) is their probability.

Type II error is the same as the type I error is the case. You begin to suppress your emotions and avoid experiencing any connection when someone thinks that you have no relation even though there does exist among you. In this sort of error, the researcher is expected to see the research hypothesis as true and treat the null hypotheses as false while he may do not and the opposite situation happens. Type II error is identified with β that equals to the possibility to make a type II error which is an error of omission.

Statistical Tests

One-tailed and two-tailed statistical tests help determine how significant a finding is in a set of data.

When we think that a parameter might change in one specific direction from a baseline, we use a one-tailed test. For example, if we’re testing whether a new drug makes people perform better, we might only care if it improves performance, not if it makes it worse.

On the flip side, a two-tailed test comes into play when changes could go in either direction from the baseline. For instance, if we’re studying the effect of a new teaching method on test scores, we’d want to know if it makes scores better or worse, so we’d use a two-tailed test.

Types of Statistical Tests

Hypothesis testing can be done via use of either one-tailed or two-tailed statistical test. The purpose of these tests is to obtain the probability with which a parameter from a given data set is statistically significant. These are also called lateral flow and dipstick tests.

• One-tailed test can be used so that the differences of the parameter estimations within only one side from a given standard can be perceived plausible.
• Two-tailed test needs to be applied in the case when you consider deviations from both sides of benchmark value as possible in science.

The expression “tail” is used in the terminology in which those tests are referred and the reason for that is that outliers, i.e. observation ended up rejecting the null hypothesis, are the extreme points of the distribution, those areas normally have a small influence or “tail off” similar to the bell shape or normal distribution. One study should make an application either the one-tailed test or two-tailed test according to the judgment of the research hypothesis.

What is p-Value Testing?

In the case of data information significance, the p-value is an additional and significant term for hypothesis testing. The p-value is a function whose domain is the observed result of sample and range is testing subset of statistical hypothesis which is being used for testing of statistical hypothesis. It must determine what the threshold value is before starting of the test. The significance level holds the name, traditional 1% or 5%, which stands for the level of the significance considered to be of value. One of the parameters of the Savings function is α.

In the condition if the p-value is greater than or equal the α term, inconsistency between our null model and the data exists. As a result the null hypothesis should be rejected and a new hypothesis may be supposed being true, or may be assumed as such one.

Example on Test of Significance

Some examples of test of significance are added below:

Example 1: T-Test for Medical Research – The T Test

For example, a medical study researching the performance of a new drug that comes to the conclusion of a reduced in blood pressure. The researchers predict that the patients taking the new drug will show a frankly larger decrease in blood pressure as opposed to the study participants on a placebo. They collect data from two groups: treat one group with an experimental drug and give all the placebo to the second group.

Researchers apply a t-test to the data in order determine the value of two assumed normal populations difference and study whether it statistically significant. The H0 (null hypothesis) could state that there is no significant difference in the blood pressure registered in the two groups of subjects, while the HA1 (alternative hypothesis) should be indicating the positivity of a significant difference. They can check whether or not the outcomes are significantly different by using the t-test, and therefore reduce the possibility of any confusing hypotheses.

Example 2: Chi-Square Analysis in Market Research

Think about the situation where you have to carry out a market research work to ascertain the link between customers satisfaction (comprised of satisfied satisfied or neutral scores) and their product preferences (the three products designated as Product A, Product B, and Product C). A chi-square test was used by the researchers to check whether they had a substantial association with the two categorical variables they were dealing with.

The H0 null hypothesis states customer satisfaction and product preferences are unrelated, the contrary to which H1 alternative hypothesis shows the customers’ satisfaction and product preferences are related. Thereby, the researchers will be able to execute the chi-square test on the gathered data and find out if the existed observations among customer satisfaction and product preferences are statistically significant by doing so. This allows us to make conclusions how the satisfaction degree of customers affects the market conception of goods for the target market.

Example 3: ANOVA in Educational Research

Think of a researcher whom is studying if there is any difference between various learning ways and their effect on students’ study achievements. HO represents the null hypothesis which asserts no differences in scores for the groups while the alternative hypothesis (HA) claims at least one group has a different mean. Via use Analysis of Variance ( ANOVA ), a researcher determines whether or not there is any statistically significant difference in performance hence, across the methods of teaching.

Example 4: Regression Analysis in Economics

In an economic study, researchers examine the connection between ads cost and revenue for the group of businesses that have recently disclosed their financial results. The null space proposes that there is no such linear connection between the advertisement spending and purchases.

Among the models, the regression analysis used to determine whether the changes in sales are attributed to the changes in advertising to a statistically significant level (the regression line slope is significantly different from zero) is chosen.

Example 5: Paired T-Test in Psychology

A psychologist decides to do a study to find out if a new type of therapy can make someone get rid of anxiety. Patients are evaluated of their level of anxiety prior to initiating the intervention and right after.

The null hypothesis claims that there is no noticeable difference in the levels of anxiety from a pre-intervention to a post-intervention setting. Using a paired t-test, a psychologist who collected the anxiety scores of a group before and after the experiment can prove statistically the observed change in these scores.

Test of Significance – FAQs

What is test of significance.

Test of significance is a process for comparing observed data with a claim(also called a hypothesis), the truth of which is being assessed in further analysis.

Define Statistical Significance Test?

Random distribution of observed data implies that there must be a certain cause behind which could then be associated with the data. This outcome is also referred to as the statistical significance. Whatever the explicit field or the profession that rely utterly on numbers and research, like finance, economics, investing, medicine, and biology, statistic is important.

What is the meaning of a test of significance?

Statistical significant tests work in order to determine if the differences found in assessment data is just due to random errors arising from sampling or not. This is a “silent” category of research that ought to be overlooked for it brings on mere incompatibilities.

What is the importance of the Significance test?

In experiments, the significance tests indeed have specific applied value. That is because they help researchers to draw conclusion whether the data supports or not the null hypothesis, and therefore whether the alternative hypothesis is true or not.

How many types of Significance tests are there in statistical mathematics?

In statistics, we have tests like t-test, aZ-test, chi-square test, annoVA test, binomial test, mediana test and others. Greatly decentralized data can be tested with parametric tests.

How does choosing a significance level (α) influence the interpretation of the attributable tests?

The parameter α which stands for the significance level is a function of this threshold, and to fail this test null hypothesis value has to be rejected. Hence, a smaller α value means higher strictness of acceptance threshold and false positives are limited while there could be an increase in false negatives.

Is significance testing limited to parametric methods like comparison of two means or, it can be applied to non-parametric datasets also?

Inference is something useful which can be miscellaneous and can adapt to parametric or non-parametric data. Non-parametric tests, for instance the Mann-Whitney U test and the Wilcoxon signed-rank test, are often applied in operations research, since they do not require that data meet the assumptions of parametric tests.

Please Login to comment...

Similar reads.

• Math-Statistics
• School Learning

Improve your Coding Skills with Practice

What kind of Experience do you want to share?

If you're seeing this message, it means we're having trouble loading external resources on our website.

If you're behind a web filter, please make sure that the domains *.kastatic.org and *.kasandbox.org are unblocked.

To log in and use all the features of Khan Academy, please enable JavaScript in your browser.

Statistics and probability

Course: statistics and probability   >   unit 12, hypothesis testing and p-values.

• One-tailed and two-tailed tests
• Z-statistics vs. T-statistics
• Small sample hypothesis test
• Large sample proportion hypothesis testing

Want to join the conversation?

• Upvote Button navigates to signup page
• Downvote Button navigates to signup page
• Flag Button navigates to signup page

Video transcript