how to write null hypothesis for one way anova

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

11.4 One-Way ANOVA and Hypothesis Tests for Three or More Population Means

Learning objectives.

• Conduct and interpret hypothesis tests for three or more population means using one-way ANOVA.

The purpose of a one-way ANOVA (analysis of variance) test is to determine the existence of a statistically significant difference among the means of three or more populations.  The test actually uses variances to help determine if the population means are equal or not.

Throughout this section, we will use subscripts to identify the values for the means, sample sizes, and standard deviations for the populations:

[latex]k[/latex] is the number of populations under study, [latex]n[/latex] is the total number of observations in all of the samples combined, and [latex]\overline{\overline{x}}[/latex] is the mean of the sample means.

[latex]\begin{eqnarray*} n & = & n_1+n_2+\cdots+n_k \\ \\ \overline{\overline{x}} & = & \frac{n_1 \times \overline{x}_1 +n_2 \times \overline{x}_2 +\cdots+n_k \times \overline{x}_k}{n} \end{eqnarray*}[/latex]

One-Way ANOVA

A predictor variable is called a factor or independent variable .  For example age, temperature, and gender are factors.  The groups or samples are often referred to as treatments .  This terminology comes from the use of ANOVA procedures in medical and psychological research to determine if there is a difference in the effects of different treatments.

A local college wants to compare the mean GPA for players on four of its sports teams:  basketball, baseball, hockey, and lacrosse.  A random sample of players was taken from each team and their GPA recorded in the table below.

In this example, the factor is the sports team.

[latex]\begin{eqnarray*} k & = & 4 \\ \\ n & = & n_1+n_2+n_3+n_4 \\ & = & 5+5+5+5 \\ & = & 20 \\ \\ \overline{\overline{x}} & = & \frac{n_1 \times \overline{x}_1+n_2 \times \overline{x}_2+n_3 \times \overline{x}_3+n_4 \times \overline{x}_4}{n} \\ & = & \frac{5 \times 3.22+5 \times 3.02+5 \times 3+5 \times 2.94}{20}  \\& = & 3.045 \end{eqnarray*}[/latex]

The following assumptions are required to use a one-way ANOVA test:

• Each population from which a sample is taken is normally distributed.
• All samples are randomly selected and independently taken from the populations.
• The populations are assumed to have equal variances.
• The population data is numerical (interval or ratio level).

The logic behind one-way ANOVA is to compare population means based on two independent estimates of the (assumed) equal variance [latex]\sigma^2[/latex] between the populations:

• One estimate of the equal variance [latex]\sigma^2[/latex] is based on the variability among the sample means themselves (called the between-groups estimate of population variance).
• One estimate of the equal variance [latex]\sigma^2[/latex] is based on the variability of the data within each sample (called the within-groups estimate of population variance).

The one-way ANOVA procedure compares these two estimates of the population variance [latex]\sigma^2[/latex] to determine if the population means are equal or if there is a difference in the population means.  Because ANOVA involves the comparison of two estimates of variance, an [latex]F[/latex]-distribution is used to conduct the ANOVA test.  The test statistic is an [latex]F[/latex]-score that is the ratio of the two estimates of population variance:

[latex]\displaystyle{F=\frac{\mbox{variance between groups}}{\mbox{variance within groups}}}[/latex]

The degrees of freedom for the [latex]F[/latex]-distribution are [latex]df_1=k-1[/latex] and [latex]df_2=n-k[/latex] where [latex]k[/latex] is the number of populations and [latex]n[/latex] is the total number of observations in all of the samples combined.

The variance between groups estimate of the population variance is called the mean square due to treatment , [latex]MST[/latex].  The [latex]MST[/latex] is the estimate of the population variance determined by the variance of the sample means from the overall sample mean [latex]\overline{\overline{x}}[/latex].  When the population means are equal, [latex]MST[/latex] provides an unbiased estimate of the population variance.  When the population means are not equal, [latex]MST[/latex] provides an overestimate of the population variance.

[latex]\begin{eqnarray*} SST & = & n_1 \times (\overline{x}_1-\overline{\overline{x}})^2+n_2\times (\overline{x}_2-\overline{\overline{x}})^2+ \cdots +n_k \times (\overline{x}_k-\overline{\overline{x}})^2 \\  \\ MST & =& \frac{SST}{k-1} \end{eqnarray*}[/latex]

The variance within groups estimate of the population variance is called the mean square due to error , [latex]MSE[/latex].  The [latex]MSE[/latex] is the pooled estimate of the population variance using the sample variances as estimates for the population variance.  The [latex]MSE[/latex] always provides an unbiased estimate of the population variance because it is not affected by whether or not the population means are equal.

[latex]\begin{eqnarray*} SSE & = & (n_1-1) \times s_1^2+ (n_2-1) \times s_2^2+ \cdots + (n_k-1) \times s_k^2\\  \\ MSE & =& \frac{SSE}{n -k} \end{eqnarray*}[/latex]

The one-way ANOVA test depends on the fact that the variance between groups [latex]MST[/latex] is influenced by differences between the population means, which results in [latex]MST[/latex] being either an unbiased or overestimate of the population variance.  Because the variance within groups [latex]MSE[/latex] compares values of each group to its own group mean, [latex]MSE[/latex] is not affected by differences between the population means and is always an unbiased estimate of the population variance.

The null hypothesis in a one-way ANOVA test is that the population means are all equal and the alternative hypothesis is that there is a difference in the population means.  The [latex]F[/latex]-score for the one-way ANOVA test is [latex]\displaystyle{F=\frac{MST}{MSE}}[/latex] with [latex]df_1=k-1[/latex] and [latex]df_2=n-k[/latex].  The p -value for the test is the area in the right tail of the [latex]F[/latex]-distribution, to the right of the [latex]F[/latex]-score.

• When the variance between groups [latex]MST[/latex] and variance within groups [latex]MSE[/latex] are close in value, the [latex]F[/latex]-score is close to 1 and results in a large p -value.  In this case, the conclusion is that the population means are equal.
• When the variance between groups [latex]MST[/latex] is significantly larger than the variability within groups [latex]MSE[/latex], the [latex]F[/latex]-score is large and results in a small p -value.  In this case, the conclusion is that there is a difference in the population means.

Steps to Conduct a Hypothesis Test for Three or More Population Means

• Verify that the one-way ANOVA assumptions are met.

[latex]\begin{eqnarray*} \\ H_0: &  &  \mu_1=\mu_2=\cdots=\mu_k\end{eqnarray*}[/latex].

[latex]\begin{eqnarray*} \\ H_a: &  & \mbox{at least one population mean is different from the others} \\ \\ \end{eqnarray*}[/latex]

• Collect the sample information for the test and identify the significance level [latex]\alpha[/latex].

[latex]\begin{eqnarray*}F & = & \frac{MST}{MSE} \\ \\ df_1 & = & k-1 \\ \\ df_2 &  = & n-k \\ \\ \end{eqnarray*}[/latex]

• The results of the sample data are significant.  There is sufficient evidence to conclude that the null hypothesis [latex]H_0[/latex] is an incorrect belief and that the alternative hypothesis [latex]H_a[/latex] is most likely correct.
• The results of the sample data are not significant.  There is not sufficient evidence to conclude that the alternative hypothesis [latex]H_a[/latex] may be correct.
• Write down a concluding sentence specific to the context of the question.

Assume the populations are normally distributed and have equal variances.  At the 5% significance level, is there a difference in the average GPA between the sports team.

Let basketball be population 1, let baseball be population 2, let hockey be population 3, and let lacrosse be population 4. From the question we have the following information:

Previously, we found [latex]k=4[/latex], [latex]n=20[/latex], and [latex]\overline{\overline{x}}=3.045[/latex].

Hypotheses:

[latex]\begin{eqnarray*} H_0: & & \mu_1=\mu_2=\mu_3=\mu_4 \\   H_a: & & \mbox{at least one population mean is different from the others} \end{eqnarray*}[/latex]

To calculate out the [latex]F[/latex]-score, we need to find [latex]MST[/latex] and [latex]MSE[/latex].

[latex]\begin{eqnarray*} SST & = & n_1 \times (\overline{x}_1-\overline{\overline{x}})^2+n_2\times (\overline{x}_2-\overline{\overline{x}})^2+n_3 \times (\overline{x}_3-\overline{\overline{x}})^2  +n_4 \times (\overline{x}_4-\overline{\overline{x}})^2\\  & = & 5 \times (3.22-3.045)^2+5 \times (3.02-3.045)^2+5 \times (3-3.045)^2 \\ &  & +5 \times (2.94 -3.045)^2 \\ & = & 0.2215 \\ \\ MST & = & \frac{SST}{k-1} \\ & = & \frac{0.2215 }{4-1} \\ & = & 0.0738...\\ \\  SSE & = & (n_1-1) \times s_1^2+ (n_2-1) \times s_2^2+  (n_3-1) \times s_3^2+ (n_4-1) \times s_4^2\\  & = &( 5-1) \times 0.277+(5-1) \times 0.487+(5-1) \times 0.56 +(5-1)\times 0.623 \\ & = & 7.788 \\ \\ MSE & = & \frac{SSE}{n-k} \\ & = & \frac{7.788 }{20-4} \\ & = & 0.48675\end{eqnarray*}[/latex]

The p -value is the area in the right tail of the [latex]F[/latex]-distribution.  To use the f.dist.rt  function, we need to calculate out the [latex]F[/latex]-score and the degrees of freedom:

[latex]\begin{eqnarray*} F & = &\frac{MST}{MSE} \\ & = & \frac{0.0738...}{0.48675} \\ & = & 0.15168... \\ \\ df_1 & = & k-1 \\ & = & 4-1 \\ & = & 3 \\ \\df_2 & = & n-k \\ & = & 20-4 \\ & = & 16\end{eqnarray*}[/latex]

So the p -value[latex]=0.9271[/latex].

Conclusion:

Because p -value[latex]=0.9271 \gt 0.05=\alpha[/latex], we do not reject the null hypothesis.  At the 5% significance level there is  enough evidence to suggest that the mean GPA for the sports teams are the same.

• The null hypothesis [latex]\mu_1=\mu_2=\mu_3=\mu_4[/latex] is the claim that the mean GPA for the sports teams are all equal.
• The alternative hypothesis is the claim that at least one of the population means is not equal to the others.  The alternative hypothesis does not say that all of the population means are not equal, only that at least one of them is not equal to the others.
• The function is f.dist.rt because we are finding the area in the right tail of an [latex]F[/latex]-distribution.
• Field 1 is the value of [latex]F[/latex].
• Field 2 is the value of [latex]df_1[/latex].
• Field 3 is the value of [latex]df_2[/latex].
• The p -value of 0.9271 is a large probability compared to the significance level, and so is likely to happen assuming the null hypothesis is true.  This suggests that the assumption that the null hypothesis is true is most likely correct, and so the conclusion of the test is to not reject the null hypothesis.  In other words, the population means are all equal.

ANOVA Summary Tables

The calculation of the [latex]MST[/latex], [latex]MSE[/latex], and the [latex]F[/latex]-score for a one-way ANOVA test can be time consuming, even with the help of software like Excel.  However, Excel has a built-in one-way ANOVA summary table that not only generates the averages, variances, [latex]MST[/latex] and [latex]MSE[/latex], but also calculates the required [latex]F[/latex]-score and p -value for the test.

USING EXCEL TO CREATE A ONE-WAY ANOVA SUMMARY TABLE

In order to create a one-way ANOVA summary table, we need to use the Analysis ToolPak.  Follow these instructions to add the Analysis ToolPak.

• Enter the data into an Excel worksheet.
• Go to the Data tab and click on Data Analysis .  If you do not see Data Analysis in the Data tab, you will need to install the Analysis ToolPak.
• In the Data Analysis window, select Anova:  Single Factor .  Click OK .
• In the Inpu t range, enter the cell range for the data.
• In the Grouped By box, select rows if your data is entered as rows (the default is columns).
• Click on Labels in first row if the you included the column headings in the input range.
• In the Alpha box, enter the significance level for the test.
• From the Output Options , select the location where you want the output to appear.

This website provides additional information on using Excel to create a one-way ANOVA summary table.

Because we are using the p -value approach to hypothesis testing, it is not crucial that we enter the actual significance level we are using for the test.  The p -value (the area in the right tail of the [latex]F[/latex]-distribution) is not affected by significance level.  For the critical-value approach to hypothesis testing, we must enter the correct significance level for the test because the critical value does depend on the significance level.

Let basketball be population 1, let baseball be population 2, let hockey be population 3, and let lacrosse be population 4.

The ANOVA summary table generated by Excel is shown below:

The p -value for the test is in the P -value column of the between groups row .  So the p -value[latex]=0.9271[/latex].

• In the top part of the ANOVA summary table (under the Summary heading), we have the averages and variances for each of the groups (basketball, baseball, hockey, and lacrosse).
• The value of [latex]SST[/latex] (in the SS column of the between groups row).
• The value of [latex]MST[/latex] (in the MS column of the between group s row).
• The value of [latex]SSE[/latex] (in the SS column of the within groups row).
• The value of [latex]MSE[/latex] (in the MS column of the within groups row).
• The value of the [latex]F[/latex]-score (in the F column of the between groups row).
• The p -value (in the p -value column of the between groups row).

A fourth grade class is studying the environment.  One of the assignments is to grow bean plants in different soils.  Tommy chose to grow his bean plants in soil found outside his classroom mixed with dryer lint.  Tara chose to grow her bean plants in potting soil bought at the local nursery.  Nick chose to grow his bean plants in soil from his mother’s garden.  No chemicals were used on the plants, only water.  They were grown inside the classroom next to a large window.  Each child grew five plants.  At the end of the growing period, each plant was measured, producing the data (in inches) in the table below.

Assume the heights of the plants are normally distribution and have equal variance.  At the 5% significance level, does it appear that the three media in which the bean plants were grown produced the same mean height?

Let Tommy’s plants be population 1, let Tara’s plants be population 2, and let Nick’s plants be population 3.

[latex]\begin{eqnarray*} H_0: & & \mu_1=\mu_2=\mu_3 \\   H_a: & & \mbox{at least one population mean is different from the others} \end{eqnarray*}[/latex]

So the p -value[latex]=0.8760[/latex].

Because p -value[latex]=0.8760 \gt 0.05=\alpha[/latex], we do not reject the null hypothesis.  At the 5% significance level there is  enough evidence to suggest that the mean heights of the plants grown in three media are the same.

• The null hypothesis [latex]\mu_1=\mu_2=\mu_3[/latex] is the claim that the mean heights of the plants grown in the three different media are all equal.
• The p -value of 0.8760 is a large probability compared to the significance level, and so is likely to happen assuming the null hypothesis is true.  This suggests that the assumption that the null hypothesis is true is most likely correct, and so the conclusion of the test is to not reject the null hypothesis.  In other words, the population means are all equal.

A statistics professor wants to study the average GPA of students in four different programs: marketing, management, accounting, and human resources.  The professor took a random sample of GPAs of students in those programs at the end of the past semester.  The data is recorded in the table below.

Assume the GPAs of the students are normally distributed and have equal variance.  At the 5% significance level, is there a difference in the average GPA of the students in the different programs?

Let marketing be population 1, let management be population 2, let accounting be population 3, and let human resources be population 4.

[latex]\begin{eqnarray*} H_0: & & \mu_1=\mu_2=\mu_3=\mu_4\\   H_a: & & \mbox{at least one population mean is different from the others} \end{eqnarray*}[/latex]

So the p -value[latex]=0.0462[/latex].

Because p -value[latex]=0.0462 \lt 0.05=\alpha[/latex], we reject the null hypothesis in favour of the alternative hypothesis.  At the 5% significance level there is enough evidence to suggest that there is a difference in the average GPA of the students in the different programs.

A manufacturing company runs three different production lines to produce one of its products.  The company wants to know if the average production rate is the same for the three lines.  For each production line, a sample of eight hour shifts was taken and the number of items produced during each shift was recorded in the table below.

Assume the numbers of items produced on each line during an eight hour shift are normally distributed and have equal variance.  At the 1% significance level, is there a difference in the average production rate for the three lines?

Let Line 1 be population 1, let Line 2 be population 2, and let Line 3 be population 3.

So the p -value[latex]=0.0073[/latex].

Because p -value[latex]=0.0073 \lt 0.01=\alpha[/latex], we reject the null hypothesis in favour of the alternative hypothesis.  At the 1% significance level there is enough evidence to suggest that there is a difference in the average production rate of the three lines.

Concept Review

A one-way ANOVA hypothesis test determines if several population means are equal.  In order to conduct a one-way ANOVA test, the following assumptions must be met:

• Each population from which a sample is taken is assumed to be normal.
• All samples are randomly selected and independent.

The analysis of variance procedure compares the variation between groups [latex]MST[/latex] to the variation within groups [latex]MSE[/latex]. The ratio of these two estimates of variance is the [latex]F[/latex]-score from an [latex]F[/latex]-distribution with [latex]df_1=k-1[/latex] and [latex]df_2=n-k[/latex].  The p -value for the test is the area in the right tail of the [latex]F[/latex]-distribution.  The statistics used in an ANOVA test are summarized in the ANOVA summary table generated by Excel.

The one-way ANOVA hypothesis test for three or more population means is a well established process:

• Write down the null and alternative hypotheses in terms of the population means.  The null hypothesis is the claim that the population means are all equal and the alternative hypothesis is the claim that at least one of the population means is different from the others.
• Collect the sample information for the test and identify the significance level.
• The p -value is the area in the right tail of the [latex]F[/latex]-distribution.  Use the ANOVA summary table generated by Excel to find the p -value.
• Compare the  p -value to the significance level and state the outcome of the test.

Attribution

“ 13.1   One-Way ANOVA “  and “ 13.2   The F Distribution and the F-Ratio “ in Introductory Statistics by OpenStax  is licensed under a  Creative Commons Attribution 4.0 International License .

Introduction to Statistics Copyright © 2022 by Valerie Watts is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

User Preferences

Content preview.

Arcu felis bibendum ut tristique et egestas quis:

• Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris
• Duis aute irure dolor in reprehenderit in voluptate
• Excepteur sint occaecat cupidatat non proident

Keyboard Shortcuts

10.2 - a statistical test for one-way anova.

Before we go into the details of the test, we need to determine the null and alternative hypotheses. Recall that for a test for two independent means, the null hypothesis was \(\mu_1=\mu_2\). In one-way ANOVA, we want to compare \(t\) population means, where \(t>2\). Therefore, the null hypothesis for analysis of variance for \(t\) population means is:

\(H_0\colon \mu_1=\mu_2=...\mu_t\)

The alternative, however, cannot be set up similarly to the two-sample case. If we wanted to see if two population means are different, the alternative would be \(\mu_1\ne\mu_2\). With more than two groups, the research question is “Are some of the means different?." If we set up the alternative to be \(\mu_1\ne\mu_2\ne…\ne\mu_t\), then we would have a test to see if ALL the means are different. This is not what we want. We need to be careful how we set up the alternative. The mathematical version of the alternative is...

\(H_a\colon \mu_i\ne\mu_j\text{ for some }i \text{ and }j \text{ where }i\ne j\)

This means that at least one of the pairs is not equal. The more common presentation of the alternative is:

\(H_a\colon \text{ at least one mean is different}\) or \(H_a\colon \text{ not all the means are equal}\)

Recall that when we compare the means of two populations for independent samples, we use a 2-sample t -test with pooled variance when the population variances can be assumed equal.

For more than two populations, the test statistic, \(F\), is the ratio of between group sample variance and the within-group-sample variance. That is,

\(F=\dfrac{\text{between group variance}}{\text{within group variance}}\)

Under the null hypothesis (and with certain assumptions), both quantities estimate the variance of the random error, and thus the ratio should be close to 1. If the ratio is large, then we have evidence against the null, and hence, we would reject the null hypothesis.

In the next section, we present the assumptions for this test. In the following section, we present how to find the between group variance, the within group variance, and the F-statistic in the ANOVA table.

JMP | Statistical Discovery.™ From SAS.

Statistics Knowledge Portal

A free online introduction to statistics

One-Way ANOVA

What is one-way anova.

One-way analysis of variance (ANOVA) is a statistical method for testing for differences in the means of three or more groups.

How is one-way ANOVA used?

One-way ANOVA is typically used when you have a single independent variable, or factor , and your goal is to investigate if variations, or different levels  of that factor have a measurable effect on a dependent variable.

What are some limitations to consider?

One-way ANOVA can only be used when investigating a single factor and a single dependent variable. When comparing the means of three or more groups, it can tell us if at least one pair of means is significantly different, but it can’t tell us which pair. Also, it requires that the dependent variable be normally distributed in each of the groups and that the variability within groups is similar across groups.

One-way ANOVA is a test for differences in group means

See how to perform a one-way anova using statistical software.

• Download JMP to follow along using the sample data included with the software.
• To see more JMP tutorials, visit the JMP Learning Library .

One-way ANOVA is a statistical method to test the null hypothesis ( H 0 ) that three or more population means are equal vs. the alternative hypothesis ( H a ) that at least one mean is different. Using the formal notation of statistical hypotheses, for k means we write:

$ H_0:\mu_1=\mu_2=\cdots=\mu_k $

$ H_a:\mathrm{not\mathrm{\ }all\ means\ are\ equal} $

where $\mu_i$ is the mean of the i -th level of the factor.

Okay, you might be thinking, but in what situations would I need to determine if the means of multiple populations are the same or different? A common scenario is you suspect that a particular independent process variable is a driver of an important result of that process. For example, you may have suspicions about how different production lots, operators or raw material batches are affecting the output (aka a quality measurement) of a production process.

To test your suspicion, you could run the process using three or more variations (aka levels) of this independent variable (aka factor), and then take a sample of observations from the results of each run. If you find differences when comparing the means from each group of observations using an ANOVA, then (assuming you’ve done everything correctly!) you have evidence that your suspicion was correct—the factor you investigated appears to play a role in the result!

A one-way ANOVA example

Let's work through a one-way ANOVA example in more detail. Imagine you work for a company that manufactures an adhesive gel that is sold in small jars. The viscosity of the gel is important: too thick and it becomes difficult to apply; too thin and its adhesiveness suffers. You've received some feedback from a few unhappy customers lately complaining that the viscosity of your adhesive is not as consistent as it used to be. You've been asked by your boss to investigate.

You decide that a good first step would be to examine the average viscosity of the five most recent production lots. If you find differences between lots, that would seem to confirm the issue is real. It might also help you begin to form hypotheses about factors that could cause inconsistencies between lots.

You measure viscosity using an instrument that rotates a spindle immersed in the jar of adhesive. This test yields a measurement called torque resistance. You test five jars selected randomly from each of the most recent five lots. You obtain the torque resistance measurement for each jar and plot the data.

From the plot of the data, you observe that torque measurements from the Lot 3 jars tend to be lower than the torque measurements from the samples taken from the other lots. When you calculate the means from all your measurements, you see that the mean torque for Lot 3 is 26.77—much lower than the other four lots, each with a mean of around 30.

Table 1: Mean torque measurements from tests of five lots of adhesive

The ANOVA table

ANOVA results are typically displayed in an ANOVA table. An ANOVA table includes:

• Source: the sources of variation including the factor being examined (in our case, lot), error and total.
• DF: degrees of freedom for each source of variation.
• Sum of Squares: sum of squares (SS) for each source of variation along with the total from all sources.
• Mean Square: sum of squares divided by its associated degrees of freedom.
• F Ratio: the mean square of the factor (lot) divided by the mean square of the error.
• Prob > F: the p-value.

Table 2: ANOVA table with results from our torque measurements

We'll explain how the components of this table are derived below. One key element in this table to focus on for now is the p-value. The p-value is used to evaluate the validity of the null hypothesis that all the means are the same. In our example, the p-value (Prob > F) is 0.0012. This small p-value can be taken as evidence that the means are not all the same. Our samples provide evidence that there is a difference in the average torque resistance values between one or more of the five lots.

What is a p-value?

A p-value is a measure of probability used for hypothesis testing. The goal of hypothesis testing is to determine whether there is enough evidence to support a certain hypothesis about your data. Recall that with ANOVA, we formulate two hypotheses: the null hypothesis that all the means are equal and the alternative hypothesis that the means are not all equal.

Because we’re only examining random samples of data pulled from whole populations, there’s a risk that the means of our samples are not representative of the means of the full populations. The p-value gives us a way to quantify that risk. It is the probability that any variability in the means of your sample data is the result of pure chance; more specifically, it’s the probability of observing variances in the sample means at least as large as what you’ve measured when in fact the null hypothesis is true (the full population means are, in fact, equal).

A small p-value would lead you to reject the null hypothesis. A typical threshold for rejection of a null hypothesis is 0.05. That is, if you have a p-value less than 0.05, you would reject the null hypothesis in favor of the alternative hypothesis that at least one mean is different from the rest.

Based on these results, you decide to hold Lot 3 for further testing. In your report you might write: The torque from five jars of product were measured from each of the five most recent production lots. An ANOVA analysis found that the observations support a difference in mean torque between lots (p = 0.0012). A plot of the data shows that Lot 3 had a lower mean (26.77) torque as compared to the other four lots. We will hold Lot 3 for further evaluation.

Remember, an ANOVA test will not tell you which mean or means differs from the others, and (unlike our example) this isn't always obvious from a plot of the data. One way to answer questions about specific types of differences is to use a multiple comparison test. For example, to compare group means to the overall mean, you can use analysis of means (ANOM). To compare individual pairs of means, you can use the Tukey-Kramer multiple comparison test.

One-way ANOVA calculation

Now let’s consider our torque measurement example in more detail. Recall that we had five lots of material. From each lot we randomly selected five jars for testing. This is called a one-factor design. The one factor, lot, has five levels. Each level is replicated (tested) five times. The results of the testing are listed below.

Table 3: Torque measurements by Lot

To explore the calculations that resulted in the ANOVA table above (Table 2), let's first establish the following definitions:

$n_i$ = Number of observations for treatment $i$ (in our example, Lot $i$)

$N$ = Total number of observations

$Y_{ij}$ = The j th observation on the i th treatment

$\overline{Y}_i$ = The sample mean for the i th treatment

$\overline{\overline{Y}}$ = The mean of all observations (grand mean)

Sum of Squares

With these definitions in mind, let's tackle the Sum of Squares column from the ANOVA table. The sum of squares gives us a way to quantify variability in a data set by focusing on the difference between each data point and the mean of all data points in that data set. The formula below partitions the overall variability into two parts: the variability due to the model or the factor levels, and the variability due to random error.  

$$ \sum_{i=1}^{a}\sum_{j=1}^{n_i}(Y_{ij}-\overline{\overline{Y}})^2\;=\;\sum_{i=1}^{a}n_i(\overline{Y}_i-\overline{\overline{Y}})^2+\sum_{i=1}^{a}\sum_{j=1}^{n_i}(Y_{ij}-\overline{Y}_i)^2 $$

$$ SS(Total)\;     =     \;SS(Factor)\;     +     \;SS(Error) $$

While that equation may seem complicated, focusing on each element individually makes it much easier to grasp. Table 4 below lists each component of the formula and then builds them into the squared terms that make up the sum of squares. The first column of data ($Y_{ij}$) contains the torque measurements we gathered in Table 3 above.

Another way to look at sources of variability: between group variation and within group variation

Recall that in our ANOVA table above (Table 2), the Source column lists two sources of variation: factor (in our example, lot) and error. Another way to think of those two sources is between group variation (which corresponds to variation due to the factor or treatment) and within group variation (which corresponds to variation due to chance or error). So using that terminology, our sum of squares formula is essentially calculating the sum of variation due to differences between the groups (the treatment effect) and variation due to differences within each group (unexplained differences due to chance).  

Table 4: Sum of squares calculation

Degrees of Freedom (DF)

Associated with each sum of squares is a quantity called degrees of freedom (DF). The degrees of freedom indicate the number of independent pieces of information used to calculate each sum of squares. For a one factor design with a factor at k levels (five lots in our example) and a total of N observations (five jars per lot for a total of 25), the degrees of freedom are as follows:

Table 5: Determining degrees of freedom

Mean Squares (MS) and F Ratio

We divide each sum of squares by the corresponding degrees of freedom to obtain mean squares. When the null hypothesis is true (i.e. the means are equal), MS (Factor) and MS (Error) are both estimates of error variance and would be about the same size. Their ratio, or the F ratio, would be close to one. When the null hypothesis is not true then the MS (Factor) will be larger than MS (Error) and their ratio greater than 1. In our adhesive testing example, the computed F ratio, 6.90, presents significant evidence against the null hypothesis that the means are equal.

Table 6: Calculating mean squares and F ratio

The ratio of MS(factor) to MS(error)—the F ratio—has an F distribution. The F distribution is the distribution of F values that we'd expect to observe when the null hypothesis is true (i.e. the means are equal). F distributions have different shapes based on two parameters, called the numerator and denominator degrees of freedom. For an ANOVA test, the numerator is the MS(factor), so the degrees of freedom are those associated with the MS(factor). The denominator is the MS(error), so the denominator degrees of freedom are those associated with the MS(error).

If your computed F ratio exceeds the expected value from the corresponding F distribution, then, assuming a sufficiently small p-value, you would reject the null hypothesis that the means are equal. The p-value in this case is the probability of observing a value greater than the F ratio from the F distribution when in fact the null hypothesis is true.

Module 13: F-Distribution and One-Way ANOVA

One-way anova, learning outcomes.

• Conduct and interpret one-way ANOVA

The purpose of a one-way ANOVA test is to determine the existence of a statistically significant difference among several group means. The test actually uses variances to help determine if the means are equal or not. In order to perform a one-way ANOVA test, there are five basic assumptions to be fulfilled:

• Each population from which a sample is taken is assumed to be normal.
• All samples are randomly selected and independent.
• The populations are assumed to have equal standard deviations (or variances) .
• The factor is a categorical variable.
• The response is a numerical variable.

The Null and Alternative Hypotheses

The null hypothesis is simply that all the group population means are the same. The alternative hypothesis is that at least one pair of means is different. For example, if there are k groups:

H 0 : μ 1 = μ 2 = μ 3 = … = μ k

H a : At least two of the group means μ 1 , μ 2 , μ 3 , …, μ k are not equal.

The graphs, a set of box plots representing the distribution of values with the group means indicated by a horizontal line through the box, help in the understanding of the hypothesis test. In the first graph (red box plots), H 0 : μ 1 = μ 2 = μ 3 and the three populations have the same distribution if the null hypothesis is true. The variance of the combined data is approximately the same as the variance of each of the populations.

If the null hypothesis is false, then the variance of the combined data is larger which is caused by the different means as shown in the second graph (green box plots).

(b) H 0 is not true. All means are not the same; the differences are too large to be due to random variation.

Concept Review

Analysis of variance extends the comparison of two groups to several, each a level of a categorical variable (factor). Samples from each group are independent, and must be randomly selected from normal populations with equal variances. We test the null hypothesis of equal means of the response in every group versus the alternative hypothesis of one or more group means being different from the others. A one-way ANOVA hypothesis test determines if several population means are equal. The distribution for the test is the F distribution with two different degrees of freedom.

Assumptions:

• The populations are assumed to have equal standard deviations (or variances).
• OpenStax, Statistics, One-Way ANOVA. Located at : http://cnx.org/contents/[email protected] . License : CC BY: Attribution
• Introductory Statistics . Authored by : Barbara Illowski, Susan Dean. Provided by : Open Stax. Located at : http://cnx.org/contents/[email protected] . License : CC BY: Attribution . License Terms : Download for free at http://cnx.org/contents/[email protected]
• Completing a simple ANOVA table. Authored by : masterskills. Located at : https://youtu.be/OXA-bw9tGfo . License : All Rights Reserved . License Terms : Standard YouTube License

One-way ANOVA

What is this test for.

The one-way analysis of variance (ANOVA) is used to determine whether there are any statistically significant differences between the means of three or more independent (unrelated) groups. This guide will provide a brief introduction to the one-way ANOVA, including the assumptions of the test and when you should use this test. If you are familiar with the one-way ANOVA, but would like to carry out a one-way ANOVA analysis, go to our guide: One-way ANOVA in SPSS Statistics .

What does this test do?

The one-way ANOVA compares the means between the groups you are interested in and determines whether any of those means are statistically significantly different from each other. Specifically, it tests the null hypothesis:

where µ = group mean and k = number of groups. If, however, the one-way ANOVA returns a statistically significant result, we accept the alternative hypothesis (H A ), which is that there are at least two group means that are statistically significantly different from each other.

At this point, it is important to realize that the one-way ANOVA is an omnibus test statistic and cannot tell you which specific groups were statistically significantly different from each other, only that at least two groups were. To determine which specific groups differed from each other, you need to use a post hoc test . Post hoc tests are described later in this guide.

When might you need to use this test?

If you are dealing with individuals, you are likely to encounter this situation using two different types of study design:

One study design is to recruit a group of individuals and then randomly split this group into three or more smaller groups (i.e., each participant is allocated to one, and only one, group). You then get each group to undertake different tasks (or put them under different conditions) and measure the outcome/response on the same dependent variable. For example, a researcher wishes to know whether different pacing strategies affect the time to complete a marathon. The researcher randomly assigns a group of volunteers to either a group that (a) starts slow and then increases their speed, (b) starts fast and slows down or (c) runs at a steady pace throughout. The time to complete the marathon is the outcome (dependent) variable. This study design is illustrated schematically in the diagram below:

When you might use this test is continued on the next page .

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base
• Null and Alternative Hypotheses | Definitions & Examples

Null & Alternative Hypotheses | Definitions, Templates & Examples

Published on May 6, 2022 by Shaun Turney . Revised on June 22, 2023.

The null and alternative hypotheses are two competing claims that researchers weigh evidence for and against using a statistical test :

• Null hypothesis ( H 0 ): There’s no effect in the population .
• Alternative hypothesis ( H a or H 1 ) : There’s an effect in the population.

Table of contents

Answering your research question with hypotheses, what is a null hypothesis, what is an alternative hypothesis, similarities and differences between null and alternative hypotheses, how to write null and alternative hypotheses, other interesting articles, frequently asked questions.

The null and alternative hypotheses offer competing answers to your research question . When the research question asks “Does the independent variable affect the dependent variable?”:

• The null hypothesis ( H 0 ) answers “No, there’s no effect in the population.”
• The alternative hypothesis ( H a ) answers “Yes, there is an effect in the population.”

The null and alternative are always claims about the population. That’s because the goal of hypothesis testing is to make inferences about a population based on a sample . Often, we infer whether there’s an effect in the population by looking at differences between groups or relationships between variables in the sample. It’s critical for your research to write strong hypotheses .

You can use a statistical test to decide whether the evidence favors the null or alternative hypothesis. Each type of statistical test comes with a specific way of phrasing the null and alternative hypothesis. However, the hypotheses can also be phrased in a general way that applies to any test.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

The null hypothesis is the claim that there’s no effect in the population.

If the sample provides enough evidence against the claim that there’s no effect in the population ( p ≤ α), then we can reject the null hypothesis . Otherwise, we fail to reject the null hypothesis.

Although “fail to reject” may sound awkward, it’s the only wording that statisticians accept . Be careful not to say you “prove” or “accept” the null hypothesis.

Null hypotheses often include phrases such as “no effect,” “no difference,” or “no relationship.” When written in mathematical terms, they always include an equality (usually =, but sometimes ≥ or ≤).

You can never know with complete certainty whether there is an effect in the population. Some percentage of the time, your inference about the population will be incorrect. When you incorrectly reject the null hypothesis, it’s called a type I error . When you incorrectly fail to reject it, it’s a type II error.

Examples of null hypotheses

The table below gives examples of research questions and null hypotheses. There’s always more than one way to answer a research question, but these null hypotheses can help you get started.

*Note that some researchers prefer to always write the null hypothesis in terms of “no effect” and “=”. It would be fine to say that daily meditation has no effect on the incidence of depression and p 1 = p 2 .

The alternative hypothesis ( H a ) is the other answer to your research question . It claims that there’s an effect in the population.

Often, your alternative hypothesis is the same as your research hypothesis. In other words, it’s the claim that you expect or hope will be true.

The alternative hypothesis is the complement to the null hypothesis. Null and alternative hypotheses are exhaustive, meaning that together they cover every possible outcome. They are also mutually exclusive, meaning that only one can be true at a time.

Alternative hypotheses often include phrases such as “an effect,” “a difference,” or “a relationship.” When alternative hypotheses are written in mathematical terms, they always include an inequality (usually ≠, but sometimes < or >). As with null hypotheses, there are many acceptable ways to phrase an alternative hypothesis.

Examples of alternative hypotheses

The table below gives examples of research questions and alternative hypotheses to help you get started with formulating your own.

Null and alternative hypotheses are similar in some ways:

• They’re both answers to the research question.
• They both make claims about the population.
• They’re both evaluated by statistical tests.

However, there are important differences between the two types of hypotheses, summarized in the following table.

To help you write your hypotheses, you can use the template sentences below. If you know which statistical test you’re going to use, you can use the test-specific template sentences. Otherwise, you can use the general template sentences.

General template sentences

The only thing you need to know to use these general template sentences are your dependent and independent variables. To write your research question, null hypothesis, and alternative hypothesis, fill in the following sentences with your variables:

Does independent variable affect dependent variable ?

• Null hypothesis ( H 0 ): Independent variable does not affect dependent variable.
• Alternative hypothesis ( H a ): Independent variable affects dependent variable.

Test-specific template sentences

Once you know the statistical test you’ll be using, you can write your hypotheses in a more precise and mathematical way specific to the test you chose. The table below provides template sentences for common statistical tests.

Note: The template sentences above assume that you’re performing one-tailed tests . One-tailed tests are appropriate for most studies.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Normal distribution
• Descriptive statistics
• Measures of central tendency
• Correlation coefficient

Methodology

• Cluster sampling
• Stratified sampling
• Types of interviews
• Cohort study
• Thematic analysis

Research bias

• Implicit bias
• Cognitive bias
• Survivorship bias
• Availability heuristic
• Nonresponse bias
• Regression to the mean

Hypothesis testing is a formal procedure for investigating our ideas about the world using statistics. It is used by scientists to test specific predictions, called hypotheses , by calculating how likely it is that a pattern or relationship between variables could have arisen by chance.

Null and alternative hypotheses are used in statistical hypothesis testing . The null hypothesis of a test always predicts no effect or no relationship between variables, while the alternative hypothesis states your research prediction of an effect or relationship.

The null hypothesis is often abbreviated as H 0 . When the null hypothesis is written using mathematical symbols, it always includes an equality symbol (usually =, but sometimes ≥ or ≤).

The alternative hypothesis is often abbreviated as H a or H 1 . When the alternative hypothesis is written using mathematical symbols, it always includes an inequality symbol (usually ≠, but sometimes < or >).

A research hypothesis is your proposed answer to your research question. The research hypothesis usually includes an explanation (“ x affects y because …”).

A statistical hypothesis, on the other hand, is a mathematical statement about a population parameter. Statistical hypotheses always come in pairs: the null and alternative hypotheses . In a well-designed study , the statistical hypotheses correspond logically to the research hypothesis.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, June 22). Null & Alternative Hypotheses | Definitions, Templates & Examples. Scribbr. Retrieved June 24, 2024, from https://www.scribbr.com/statistics/null-and-alternative-hypotheses/

Is this article helpful?

Shaun Turney

Other students also liked, inferential statistics | an easy introduction & examples, hypothesis testing | a step-by-step guide with easy examples, type i & type ii errors | differences, examples, visualizations, what is your plagiarism score.

Hypothesis Testing - Analysis of Variance (ANOVA)

•   1  
• |   2  
• |   3  
• |   4  
• |   5  

The ANOVA Approach

Test statistic for anova.

All Modules

Table of F-Statistic Values

Consider an example with four independent groups and a continuous outcome measure. The independent groups might be defined by a particular characteristic of the participants such as BMI (e.g., underweight, normal weight, overweight, obese) or by the investigator (e.g., randomizing participants to one of four competing treatments, call them A, B, C and D). Suppose that the outcome is systolic blood pressure, and we wish to test whether there is a statistically significant difference in mean systolic blood pressures among the four groups. The sample data are organized as follows:

The hypotheses of interest in an ANOVA are as follows:

• H 0 : μ 1 = μ 2 = μ 3 ... = μ k
• H 1 : Means are not all equal.

where k = the number of independent comparison groups.

In this example, the hypotheses are:

• H 0 : μ 1 = μ 2 = μ 3 = μ 4
• H 1 : The means are not all equal.

The null hypothesis in ANOVA is always that there is no difference in means. The research or alternative hypothesis is always that the means are not all equal and is usually written in words rather than in mathematical symbols. The research hypothesis captures any difference in means and includes, for example, the situation where all four means are unequal, where one is different from the other three, where two are different, and so on. The alternative hypothesis, as shown above, capture all possible situations other than equality of all means specified in the null hypothesis.

The test statistic for testing H 0 : μ 1 = μ 2 = ... =   μ k is:

and the critical value is found in a table of probability values for the F distribution with (degrees of freedom) df 1 = k-1, df 2 =N-k. The table can be found in "Other Resources" on the left side of the pages.

NOTE: The test statistic F assumes equal variability in the k populations (i.e., the population variances are equal, or s 1 2 = s 2 2 = ... = s k 2 ). This means that the outcome is equally variable in each of the comparison populations. This assumption is the same as that assumed for appropriate use of the test statistic to test equality of two independent means. It is possible to assess the likelihood that the assumption of equal variances is true and the test can be conducted in most statistical computing packages. If the variability in the k comparison groups is not similar, then alternative techniques must be used.

The F statistic is computed by taking the ratio of what is called the "between treatment" variability to the "residual or error" variability. This is where the name of the procedure originates. In analysis of variance we are testing for a difference in means (H 0 : means are all equal versus H 1 : means are not all equal) by evaluating variability in the data. The numerator captures between treatment variability (i.e., differences among the sample means) and the denominator contains an estimate of the variability in the outcome. The test statistic is a measure that allows us to assess whether the differences among the sample means (numerator) are more than would be expected by chance if the null hypothesis is true. Recall in the two independent sample test, the test statistic was computed by taking the ratio of the difference in sample means (numerator) to the variability in the outcome (estimated by Sp).  

The decision rule for the F test in ANOVA is set up in a similar way to decision rules we established for t tests. The decision rule again depends on the level of significance and the degrees of freedom. The F statistic has two degrees of freedom. These are denoted df 1 and df 2 , and called the numerator and denominator degrees of freedom, respectively. The degrees of freedom are defined as follows:

df 1 = k-1 and df 2 =N-k,

where k is the number of comparison groups and N is the total number of observations in the analysis.   If the null hypothesis is true, the between treatment variation (numerator) will not exceed the residual or error variation (denominator) and the F statistic will small. If the null hypothesis is false, then the F statistic will be large. The rejection region for the F test is always in the upper (right-hand) tail of the distribution as shown below.

Rejection Region for F   Test with a =0.05, df 1 =3 and df 2 =36 (k=4, N=40)

For the scenario depicted here, the decision rule is: Reject H 0 if F > 2.87.

return to top | previous page | next page

Content ©2019. All Rights Reserved. Date last modified: January 23, 2019. Wayne W. LaMorte, MD, PhD, MPH

Teach yourself statistics

One-Way Analysis of Variance (ANOVA)

Researchers use one-way analysis of variance in controlled experiments to test for significant differences among group means. This lesson explains when, why, and how to use one-way analysis of variance. The discussion covers fixed-effects models and random-effects models .

Note: One-way analysis of variance is also known as simple analysis of variance or as single-factor analysis of variance.

When to Use One-Way ANOVA

You should only use one-way analysis of variance when you have the right data from the right experimental design .

Experimental Design

One-way analysis of variance should only be used with one type of experimental design - a completely randomized design with one factor (also known as a single-factor, independent groups design). This design is distinguished by the following attributes:

• The design has one, and only one, factor (i.e., one independent variable ) with two or more levels .
• Treatment groups are defined by a unique combination of non-overlapping factor levels.
• The design has k treatment groups, where k is greater than one.
• Experimental units are randomly selected from a known population .
• Each experimental unit is randomly assigned to one, and only one, treatment group.
• Each experimental unit provides one dependent variable score.

Data Requirements

One-way analysis of variance requires that the dependent variable be measured on an interval scale or a ratio scale . In addition, you need to know three things about the experimental design:

• k = Number of treatment groups
• n j = Number of subjects assigned to Group j (i.e., number of subjects that receive treatment j )
• X i,j = The dependent variable score for the i th subject in Group j

For example, the table below shows the critical information that a researcher would need to conduct a one-way analysis of variance, given a typical single-factor, independent groups design:

The design has three treatment groups ( k =3). Nine subjects have been randomly assigned to the groups: three subjects to Group 1 ( n 1 = 3), two subjects to Group 2 ( n 2  = 2), and four subjects to Group 3 ( n 3  = 4). The dependent variable score is X 1,1 for the first subject in Group 1; X 1,2 for the first subject in Group 2; X 1,3 for the first subject in Group 3; X 2,1 for the second subject in Group 1; and so on.

Assumptions of ANOVA

One-way analysis of variance makes three assumptions about dependent variable scores:

• Independence . The dependent variable score for each experimental unit is independent of the score for any other unit.
• Normality . In the population, dependent variable scores are normally distributed within treatment groups.
• Equality of variance . In the population, the variance of dependent variable scores in each treatment group is equal. (Equality of variance is also known as homogeneity of variance or homoscedasticity.)

The assumption of independence is the most important assumption. When that assumption is violated, the resulting statistical tests can be misleading. This assumption is tenable when (a) experimental units are randomly sampled from the population and (b) sampled units are randomly assigned to treatments.

With respect to the other two assumptions, analysis of variance is more forgiving. Violations of normality are less problematic when the sample size is large. And violations of the equal variance assumption are less problematic when the sample size within groups is equal.

Before conducting an analysis of variance, it is best practice to check for violations of normality and homogeneity assumptions. For further information, see:

• How to Test for Normality: Three Simple Tests
• How to Test for Homogeneity of Variance: Hartley's Fmax Test
• How to Test for Homogeneity of Variance: Bartlett's Test

Why to Use One-Way ANOVA

Researchers use one-way analysis of variance to assess the effect of one independent variable on one dependent variable. The analysis answers two research questions:

• Is the mean score in any treatment group significantly different from the mean score in another treatment group?
• What is the magnitude of the effect of the independent variable on the dependent variable?

Notice that analysis of variance tells us whether treatment groups differ significantly, but it doesn't tell us how the groups differ. Understanding how the groups differ requires additional analysis.

How to Use One-Way ANOVA

To implement one-way analysis of variance with a single-factor, independent groups design, a researcher takes the following steps:

• Specify a mathematical model to describe the causal factors that affect the dependent variable.
• Write statistical hypotheses to be tested by experimental data.
• Specify a significance level for a hypothesis test.
• Compute the grand mean and the mean scores for each group.
• Compute sums of squares for each effect in the model.
• Find the degrees of freedom associated with each effect in the model.
• Based on sums of squares and degrees of freedom, compute mean squares for each effect in the model.
• Find the expected value of the mean squares for each effect in the model.
• Compute a test statistic , based on observed mean squares and their expected values.
• Find the P value for the test statistic.
• Accept or reject the null hypothesis , based on the P value and the significance level.
• Assess the magnitude of the effect of the independent variable, based on sums of squares.

Whew! Altogether, the steps to implement one-way analysis of variance may look challenging, but each step is simple and logical. That makes the whole process easy to implement, if you just focus on one step at a time. So let's go over each step, one-by-one.

Mathematical Model

For every experimental design, there is a mathematical model that accounts for all of the independent and extraneous variables that affect the dependent variable.

Fixed Effects

For example, here is the fixed-effects mathematical model for a completely randomized design:

X i j = μ + β j + ε i ( j )

where X i j is the dependent variable score for subject i in treatment group j , μ is the population mean, β j is the treatment effect in group j ; and ε i ( j ) is the effect of all other extraneous variables on subject i in treatment j .

For this model, it is assumed that ε i ( j ) is normally and independently distributed with a mean of zero and a variance of σ ε 2 . The mean ( μ ) is constant.

Note: The parentheses in ε i ( j ) indicate that subjects are nested under treatment groups. When a subject is assigned to only one treatment group, we say that the subject is nested under a treatment.

Random Effects

The random-effects mathematical model for a completely randomized design is similar to the fixed-effects mathematical model. It can also be expressed as:

Like the fixed-effects mathematical model, the random-effects model also assumes that (1) ε i ( j ) is normally and independently distributed with a mean of zero and a variance of σ ε 2 and (2) the mean ( μ ) is constant.

Here's the difference between the two mathematical models. With a fixed-effects model, the experimenter includes all treatment levels of interest in the experiment. With a random-effects model, the experimenter includes a random sample of treatment levels in the experiment. Therefore, in the random-effects mathematical model, the treatment effect ( β j  ) is a random variable with a mean of zero and a variance of σ 2 β .

Statistical Hypotheses

For fixed-effects models, it is common practice to write statistical hypotheses in terms of the treatment effect β j ; for random-effects models, in terms of the treatment variance σ 2 β  .

H 0 : β j = 0 for all j (fixed-effects)

H 0 : σ 2 β = 0 for all j (random-effects)

H 1 : β j ≠ 0 for some j (fixed-effects)

H 0 : σ 2 β ≠ 0 for all j (random-effects)

If the null hypothesis is true, the mean score in each treatment group should equal the population mean. Thus, if the null hypothesis is true, sample means in the k treatment groups should be roughly equal. If the null hypothesis is false, at least one pair of sample means should be unequal.

Significance Level

The significance level (also known as alpha or α) is the probability of rejecting the null hypothesis when it is actually true. The significance level for an experiment is specified by the experimenter, before data collection begins. Experimenters often choose significance levels of 0.05 or 0.01.

A significance level of 0.05 means that there is a 5% chance of rejecting the null hypothesis when it is true. A significance level of 0.01 means that there is a 1% chance of rejecting the null hypothesis when it is true. The lower the significance level, the more persuasive the evidence needs to be before an experimenter can reject the null hypothesis.

Mean Scores

Analysis of variance begins by computing a grand mean and group means:

• Grand mean. The grand mean ( X ) is the mean of all observations, computed as follows: n = k Σ j=1 n  j X  = ( 1 / n ) k Σ j=1 n j Σ i=1 ( X  i j  )
• Group means. The mean of group j ( X j ) is the mean of all observations in group j , computed as follows:

In the equations above, n is the total sample size across all groups; and n  j is the sample size in Group j  .

Sums of Squares

A sum of squares is the sum of squared deviations from a mean score. One-way analysis of variance makes use of three sums of squares:

• Between-groups sum of squares. The between-groups sum of squares (SSB) measures variation of group means around the grand mean. It can be computed from the following formula: SSB = k Σ j=1 n j Σ i=1 (  X   j  -  X  ) 2  =  k Σ j=1 n j  (  X   j  -  X  ) 2
• Within-groups sum of squares. The within-groups sum of squares (SSW) measures variation of all scores around their respective group means. It can be computed from the following formula: SSW = k Σ j=1 n j Σ i=1 ( X  i j  -  X   j  ) 2
• Total sum of squares. The total sum of squares (SST) measures variation of all scores around the grand mean. It can be computed from the following formula: SST = k Σ j=1 n j Σ i=1 ( X  i j  -  X  ) 2

It turns out that the total sum of squares is equal to the between-groups sum of squares plus the within-groups sum of squares, as shown below:

SST = SSB + SSW

As you'll see later on, this relationship will allow us to assess the magnitude of the effect of the independent variable on the dependent variable.

Degrees of Freedom

The term degrees of freedom (df) refers to the number of independent sample points used to compute a statistic minus the number of parameters estimated from the sample points.

To illustrate what is going on, let's find the degrees of freedom associated with the various sum of squares computations:

Here, the formula uses k independent sample points, the sample means X   j  . And it uses one parameter estimate, the grand mean X , which was estimated from the sample points. So, the between-groups sum of squares has k - 1 degrees of freedom.

Here, the formula uses n independent sample points, the individual subject scores X  i j  . And it uses k parameter estimates, the group means X   j  , which were estimated from the sample points. So, the between-groups sum of squares has n - k degrees of freedom (where n is total sample size across all groups).

Here, the formula uses n independent sample points, the individual subject scores X  i j  . And it uses one parameter estimate, the grand mean X , which was estimated from the sample points. So, the total sum of squares has n  - 1 degrees of freedom (where n is total sample size across all groups).

The degrees of freedom for each sum of squares are summarized in the table below:

Notice that there is an additive relationship between the various sums of squares. The degrees of freedom for total sum of squares (df TOT ) is equal to the degrees of freedom for between-groups sum of squares (df BG ) plus the degrees of freedom for within-groups sum of squares (df WG ). That is,

df TOT = df BG + df WG

Mean Squares

A mean square is an estimate of population variance. It is computed by dividing a sum of squares (SS) by its corresponding degrees of freedom (df), as shown below:

MS = SS / df

To conduct a one-way analysis of variance, we are interested in two mean squares:

MS WG = SSW / df WG

MS BG = SSB / df BG

Expected Value

The expected value of a mean square is the average value of the mean square over a large number of experiments.

Statisticians have derived formulas for the expected value of the within-groups mean square ( MS WG  ) and for the expected value of the between-groups mean square ( MS BG  ). For one-way analysis of variance, the expected value formulas are:

Fixed- and Random-Effects:

E( MS WG  ) = σ ε 2

Fixed-Effects:

Random-Effects:

E( MS BG  ) = σ ε 2 + nσ β 2

In the equations above, E( MS WG  ) is the expected value of the within-groups mean square; E( MS BG  ) is the expected value of the between-groups mean square; n is total sample size; k is the number of treatment groups; β  j is the treatment effect in Group j ; σ ε 2 is the variance attributable to everything except the treatment effect (i.e., all the extraneous variables); and σ β 2 is the variance due to random selection of treatment levels.

Notice that MS BG should equal MS WG when the variation due to treatment effects ( β  j for fixed effects and σ β 2 for random effects) is zero (i.e., when the independent variable does not affect the dependent variable). And MS BG should be bigger than the MS WG when the variation due to treatment effects is not zero (i.e., when the independent variable does affect the dependent variable)

Conclusion: By examining the relative size of the mean squares, we can make a judgment about whether an independent variable affects a dependent variable.

Test Statistic

Suppose we use the mean squares to define a test statistic F as follows:

F(v 1 , v 2 ) = MS BG / MS WG

where MS BG is the between-groups mean square, MS WG is the within-groups mean square, v 1 is the degrees of freedom for MS BG , and v 2 is the degrees of freedom for MS WG .

Defined in this way, the F ratio measures the size of MS BG relative to MS WG . The F ratio is a convenient measure that we can use to test the null hypothesis. Here's how:

• When the F ratio is close to one, MS BG is approximately equal to MS WG . This indicates that the independent variable did not affect the dependent variable, so we cannot reject the null hypothesis.
• When the F ratio is significantly greater than one, MS BG is bigger than MS WG . This indicates that the independent variable did affect the dependent variable, so we must reject the null hypothesis.

What does it mean for the F ratio to be significantly greater than one? To answer that question, we need to talk about the P-value.

Note: With a completely randomized design, the test statistic F is computed in the same way for fixed-effects and for random-effects. With more complex designs (i.e., designs with more than one factor), test statistics may be computed differently for fixed-effects models than for random-effects models.

In an experiment, a P-value is the probability of obtaining a result more extreme than the observed experimental outcome, assuming the null hypothesis is true.

With analysis of variance, the F ratio is the observed experimental outcome that we are interested in. So, the P-value would be the probability that an F statistic would be more extreme (i.e., bigger) than the actual F ratio computed from experimental data.

How does an experimenter attach a probability to an observed F ratio? Luckily, the F ratio is a random variable that has an F distribution . Therefore, we can use an F table or an online calculator to find the probability that an F statistic will be bigger than the actual F ratio observed in the experiment.

F Distribution Calculator

To find the P-value associated with an observed F ratio, use Stat Trek's free F distribution calculator . You can access the calculator by clicking a link in the table of contents (at the top of this web page in the left column). find the calculator in the Appendix section of the table of contents, which can be accessed by tapping the "Analysis of Variance: Table of Contents" button at the top of the page. Or you can click tap the button below.

For an example that shows how to find the P-value for an F ratio, see Problem 2 at the bottom of this page.

Hypothesis Test

Recall that the experimenter specified a significance level early on - before the first data point was collected. Once you know the significance level and the P-value, the hypothesis test is routine. Here's the decision rule for accepting or rejecting the null hypothesis:

• If the P-value is bigger than the significance level, accept the null hypothesis.
• If the P-value is equal to or smaller than the significance level, reject the null hypothesis.

A "big" P-value indicates that (1) none of the k treatment means ( X j ) were significantly different, so (2) the independent variable did not have a statistically significant effect on the dependent variable.

A "small" P-value indicates that (1) at least one treatment mean differed significantly from another treatment mean, so (2) the independent variable had a statistically significant effect on the dependent variable.

Magnitude of Effect

The hypothesis test tells us whether the independent variable in our experiment has a statistically significant effect on the dependent variable, but it does not address the magnitude (strength) of the effect. Here's the issue:

• When the sample size is large, you may find that even small differences in treatment means are statistically significant.
• When the sample size is small, you may find that even big differences in treatment means are not statistically significant.

With this in mind, it is customary to supplement analysis of variance with an appropriate measure of effect size. Eta squared (η 2 ) is one such measure. Eta squared is the proportion of variance in the dependent variable that is explained by a treatment effect. The eta squared formula for one-way analysis of variance is:

η 2 = SSB / SST

where SSB is the between-groups sum of squares and SST is the total sum of squares.

ANOVA Summary Table

It is traditional to summarize ANOVA results in an analysis of variance table. Here, filled with hypothetical data, is an analysis of variance table for a one-way analysis of variance.

Analysis of Variance Table

This is an ANOVA table for a single-factor, independent groups design. The experiment used 11 treatment groups, so k equals 11. And three subjects were assigned to each treatment group, so N equals 33. The table shows critical outputs for between-group (BG) treatment effects and within-group (WG) treatment effects.

Many of the table entries are derived from the sum of squares (SS) and degrees of freedom (df), based on the following formulas:

SS TOTAL = SS BG + SS WG = 230 + 220 = 450

MS BG = SS BG / df BG = 230/10 = 23

MS WG = MS WG / df WG = 220/22 = 10

F(v 1 , v 2 ) = MS BG / MS WG = 23/10 = 2.3

where MS bg is the between-groups mean square, MS wg is the within-groups mean square, v 1 and df BG are the degrees of freedom for MS BG , v 2 and df WG are the degrees of freedom for MS WG , and the F ratio is F(v 1 , v 2 ).

An ANOVA table provides all the information an experimenter needs to (1) test hypotheses and (2) assess the magnitude of treatment effects.

Hypothesis Tests

The P-value (shown in the last column of the ANOVA table) is the probability that an F statistic would be more extreme (bigger) than the F ratio shown in the table, assuming the null hypothesis is true. When the P-value is bigger than the significance level, we accept the null hypothesis; when it is smaller, we reject it.

Suppose the significance level for this experiment was 0.05. Based on the table entries, can we reject the null hypothesis? From the ANOVA table, we see that the P-value is 0.09. Since P-value is bigger than the significance level (0.05), we cannot reject the null hypothesis.

Magnitude of Effects

Since the P-value in the ANOVA table was bigger than the significance level, the treatment effect in this experiment was not statistically significant. Does that mean the treatment effect was small? Not necessarily.

To assess the strength of the treatment effect, an experimenter might compute eta squared (η 2 ). The computation is easy, using sums of squares entries from the ANOVA table, as shown below:

η 2 = SSB / SST = 230 / 450 = 0.51

For this experiment, eta squared is 0.51. This means that 51% of the variance in the dependent variable can be explained by the effect of the independent variable.

Even though the treatment effect was not statistically significant, it was not unimportant; since the independent variable accounted for more than half the variance in the dependent variable. The moral here is that a hypothesis test by itself may not tell the whole story. It also pays to look at the magnitude of an effect.

Advantages and Disadvantages

One-way analysis of variance with a single-factor, independent groups design has advantages and disadvantages. Advantages include the following:

• The design layout is simple - one factor with k factor levels.
• Data analysis is easier with this design than with other designs.
• Computational procedures are identical for fixed-effects and random-effects models.
• The design does not require equal sample sizes for treatment groups.
• The design requires subjects to participate in only one treatment group.

Disadvantages include the following:

• The design does not permit repeated measures.
• The design can test the effect of only one independent variable.

Test Your Understanding

In analysis of variance, what is a mean square?

(A) The average deviation from the mean. (B) A measure of standard deviation. (C) A measure of variance. (D) A measure of skewness. (E) A vicious geometric shape.

The correct answer is (C). Mean squares are estimates of variance within groups or across groups. Mean squares are used to calculate F ratios, such as the following:

F = MS bg / MS wg

where MS bg is the between-group mean square and MS wg is the within-group mean square.

In the ANOVA table shown below, the P-value is missing. What is the correct entry for the P-value?

Hint: Stat Trek's F Distribution Calculator may be helpful.

(A) 0.01 (B) 0.03 (C) 0.11 (D) 0.89 (E) 0.97

The correct answer is (B).

A P-value is the probability of obtaining a result more extreme (bigger) than the observed F ratio, assuming the null hypothesis is true. From the ANOVA table, we know the following:

• The observed value of the F ratio is 3.
• The degrees of freedom (v 1 ) for the between-groups mean square is 5.
• The degrees of freedom (v 2 ) for the within-groups mean square is 30.

Therefore, the P-value we are looking for is the probability that an F with 5 and 30 degrees of freedom is greater than 3. We want to know:

P [ F(5, 30) > 3 ]

Now, we are ready to use the F Distribution Calculator . We enter the degrees of freedom (v1 = 5) for the between-groups mean square, the degrees of freedom (v2 = 30) for the within-groups mean square, and the F ratio (3) into the calculator; and hit the Calculate button.

The calculator reports that the probability that F is greater (more extreme) than 3 equals about 0.026. Hence, the correct P-value is 0.026.

13.1 One-Way ANOVA

The purpose of a one-way ANOVA test is to determine the existence of a statistically significant difference among several group means. The test uses variances to help determine if the means are equal or not. To perform a one-way ANOVA test, there are five basic assumptions to be fulfilled:

• Each population from which a sample is taken is assumed to be normal.
• All samples are randomly selected and independent.
• The populations are assumed to have equal standard deviations (or variances).
• The factor is a categorical variable.
• The response is a numerical variable.

The Null and Alternative Hypotheses

The null hypothesis is that all the group population means are the same. The alternative hypothesis is that at least one pair of means is different. For example, if there are k groups

H 0 : μ 1 = μ 2 = μ 3 = ... = μ k

H a : At least two of the group means μ 1 , μ 2 , μ 3 , ..., μ k are not equal. That is, μ i ≠ μ j for some i ≠ j .

The graphs, a set of box plots representing the distribution of values with the group means indicated by a horizontal line through the box, help in the understanding of the hypothesis test. In the first graph (red box plots), H 0 : μ 1 = μ 2 = μ 3 and the three populations have the same distribution if the null hypothesis is true. The variance of the combined data is approximately the same as the variance of each of the populations.

If the null hypothesis is false, then the variance of the combined data is larger, which is caused by the different means as shown in the second graph (green box plots).

As an Amazon Associate we earn from qualifying purchases.

This book may not be used in the training of large language models or otherwise be ingested into large language models or generative AI offerings without OpenStax's permission.

Want to cite, share, or modify this book? This book uses the Creative Commons Attribution License and you must attribute Texas Education Agency (TEA). The original material is available at: https://www.texasgateway.org/book/tea-statistics . Changes were made to the original material, including updates to art, structure, and other content updates.

Access for free at https://openstax.org/books/statistics/pages/1-introduction

• Authors: Barbara Illowsky, Susan Dean
• Publisher/website: OpenStax
• Book title: Statistics
• Publication date: Mar 27, 2020
• Location: Houston, Texas
• Book URL: https://openstax.org/books/statistics/pages/1-introduction
• Section URL: https://openstax.org/books/statistics/pages/13-1-one-way-anova

© Jan 23, 2024 Texas Education Agency (TEA). The OpenStax name, OpenStax logo, OpenStax book covers, OpenStax CNX name, and OpenStax CNX logo are not subject to the Creative Commons license and may not be reproduced without the prior and express written consent of Rice University.

Interpret the key results for One-Way ANOVA

In this topic, step 1: determine whether the differences between group means are statistically significant, step 2: examine the group means, step 3: compare the group means, step 4: determine how well the model fits your data, step 5: determine whether your model meets the assumptions of the analysis, analysis of variance.

Key Result: P-Value

In these results, the null hypothesis states that the mean hardness values of 4 different paints are equal. Because the p-value is less than the significance level of 0.05, you can reject the null hypothesis and conclude that some of the paints have different means.

Use the interval plot to display the mean and confidence interval for each group.

• Each dot represents a sample mean.
• Each interval is a 95% confidence interval for the mean of a group. You can be 95% confident that a group mean is within the group's confidence interval.

Interpret these intervals carefully because making multiple comparisons increases the type 1 error rate. That is, when you increase the number of comparisons, you also increase the probability that at least one comparison will incorrectly conclude that one of the observed differences is significantly different.

To assess the differences that appear on this plot, use the grouping information table and other comparisons output (shown in step 3).

In the interval plot, Blend 2 has the lowest mean and Blend 4 has the highest. You cannot determine from this graph whether any differences are statistically significant. To determine statistical significance, assess the confidence intervals for the differences of means.

If your one-way ANOVA p-value is less than your significance level, you know that some of the group means are different, but not which pairs of groups. Use the grouping information table and tests for differences of means to determine whether the mean difference between specific pairs of groups are statistically significant and to estimate by how much they are different.

For more information on comparison methods, go to Using multiple comparisons to assess the practical and statistical significance .

Use the grouping information table to quickly determine whether the mean difference between any pair of groups is statistically significant.

Groups that do not share a letter are significantly different.

Use the confidence intervals to determine likely ranges for the differences and to determine whether the differences are practically significant. The table displays a set of confidence intervals for the difference between pairs of means. The interval plot for differences of means displays the same information.

Confidence intervals that do not contain zero indicate a mean difference that is statistically significant.

Individual confidence level

The percentage of times that a single confidence interval includes the true difference between one pair of group means, if you repeat the study multiple times.

Simultaneous confidence level

The percentage of times that a set of confidence intervals includes the true differences for all group comparisons, if you repeat the study multiple times.

Controlling the simultaneous confidence level is particularly important when you perform multiple comparisons. If you do not control the simultaneous confidence level, the chance that at least one confidence interval does not contain the true difference increases with the number of comparisons.

For more information, go to Understanding individual and simultaneous confidence levels in multiple comparisons .

For more information about how to interpret the results for Hsu's MCB, go to What is Hsu's multiple comparisons with the best (MCB)?

Grouping Information Using the Tukey Method and 95% Confidence

Key Results: Mean, Grouping

In these results, the table shows that group A contains Blends 1, 3, and 4, and group B contains Blends 1, 2, and 3. Blends 1 and 3 are in both groups. Differences between means that share a letter are not statistically significant. Blends 2 and 4 do not share a letter, which indicates that Blend 4 has a significantly higher mean than Blend 2.

Tukey Simultaneous Tests for Differences of Means

Key Results: Simultaneous 95% CIs, Individual confidence level

• The confidence interval for the difference between the means of Blend 2 and 4 is 3.11 to 15.89. This range does not include zero, which indicates that the difference is statistically significant.
• The confidence intervals for the remaining pairs of means all include zero, which indicates that the differences are not statistically significant.
• The 95% simultaneous confidence level indicates that you can be 95% confident that all the confidence intervals contain the true differences.
• The table indicates that the individual confidence level is 98.89%. This result indicates that you can be 98.89% confident that each individual interval contains the true difference between a specific pair of group means. The individual confidence levels for each comparison produce the 95% simultaneous confidence level for all six comparisons.

To determine how well the model fits your data, examine the goodness-of-fit statistics in the Model Summary table.

S is measured in the units of the response variable and represents how far the data values fall from the fitted values. The lower the value of S, the better the model describes the response. However, a low S value by itself does not indicate that the model meets the model assumptions. You should check the residual plots to verify the assumptions.

R 2 is the percentage of variation in the response that is explained by the model. The higher the R 2 value, the better the model fits your data. R 2 is always between 0% and 100%.

A high R 2 value does not indicate that the model meets the model assumptions. You should check the residual plots to verify the assumptions.

Use predicted R 2 to determine how well your model predicts the response for new observations. Models that have larger predicted R 2 values have better predictive ability.

A predicted R 2 that is substantially less than R 2 may indicate that the model is over-fit. An over-fit model occurs when you add terms for effects that are not important in the population. The model becomes tailored to the sample data and, therefore, may not be useful for making predictions about the population.

Predicted R 2 can also be more useful than adjusted R 2 for comparing models because it is calculated with observations that are not included in the model calculation.

Model Summary

Key Results: S, R-sq, R-sq (pred)

In these results, the factor explains 47.44% of the variation in the response. S indicates that the standard deviation between the data points and the fitted values is approximately 3.95 units.

Use the residual plots to help you determine whether the model is adequate and meets the assumptions of the analysis. If the assumptions are not met, the model may not fit the data well and you should use caution when you interpret the results.

Residuals versus fits plot

Use the residuals versus fits plot to verify the assumption that the residuals are randomly distributed and have constant variance. Ideally, the points should fall randomly on both sides of 0, with no recognizable patterns in the points.

Residuals versus order plot

Normal probability plot of the residuals

Use the normal probability plot of the residuals to verify the assumption that the residuals are normally distributed. The normal probability plot of the residuals should approximately follow a straight line.

If your one-way ANOVA design meets the guidelines for sample size , the results are not substantially affected by departures from normality.

• Minitab.com
• License Portal
• Cookie Settings

You are now leaving support.minitab.com.

Click Continue to proceed to:

• Skip to secondary menu
• Skip to main content
• Skip to primary sidebar

Statistics By Jim

Making statistics intuitive

Kruskal Wallis Test Explained

By Jim Frost 2 Comments

What is the Kruskal Wallis Test?

The Kruskal Wallis test is a nonparametric hypothesis test that compares three or more independent groups. Statisticians also refer to it as one-way ANOVA on ranks. This analysis extends the Mann Whitney U nonparametric test that can compare only two groups.

If you analyze data, chances are you’re familiar with one-way ANOVA that compares the means of at least three groups. The Kruskal Wallis test is the nonparametric version of it. Because it is nonparametric, the analysis makes fewer assumptions about your data than its parametric equivalent.

Many analysts use the Kruskal Wallis test to determine whether the medians of at least three groups are unequal. However, it’s important to note that it only assesses the medians in particular circumstances. Interpreting the analysis results can be thorny. More on this later!

If you need a nonparametric test for paired groups or a single sample , consider the Wilcoxon signed rank test .

Learn more about Parametric vs. Nonparametric Tests and Hypothesis Testing Overview .

What Does the Kruskal Wallis Test Tell You?

At its core, the Kruskal Wallis test evaluates data ranks. The procedure ranks all the sample data from low to high. Then it averages the ranks for all groups. If the results are statistically significant, the average group ranks are not all equal. Consequently, the analysis indicates whether any groups have values that rank differently. For instance, one group might have values that tend to rank higher than the other groups.

The Kruskal Wallis test doesn’t involve medians or other distributional properties—just the ranks. In fact, by evaluating ranks, it rolls up both the location and shape parameters into a single evaluation of each group’s average rank.

When their average ranks are unequal, you know a group’s distribution tends to produce higher or lower values than the others. However, you don’t know enough to draw conclusions specifically about the distributions’ locations (e.g., the medians).

Special Case for Same Shapes

However, when you hold the distribution shapes constant, the Kruskal Wallis test does tell us about the median. That’s not a property of the procedure itself but logic. If several distributions have the same shape, but the average ranks are shifted higher and lower, their medians must differ. But we can only draw that conclusion about the medians when the distributions have the same shapes.

These three distributions have the same shape, but the red and green are shifted right to higher values. Wherever the median falls on the blue distribution, it’ll be in the corresponding position in the red and blue distributions. In this case, the analysis can assess the medians.

But, if the shapes aren’t similar, we don’t know whether the location, shape, or a combination of the two produced the statistically significant Kruskal Wallis test.

Analysis Assumptions

Like all statistical analyses, the Kruskal Wallis test has assumptions. Ensuring that your data meet these assumptions is crucial.

• Independent Groups : Each group has a distinct set of subjects or items.
• Independence of Observations : Each observation must be independent of the others. The data points should not influence or predict each other.
• Ordinal or Continuous Data : The Kruskal Wallis test can handle both ordinal data and continuous data, making it flexible for various research situations.
• Same Distribution Shape : This assumption applies only when you want to draw inferences about the medians. If this assumption holds, the analysis can provide insights about the medians.

Violating these assumptions can lead to incorrect conclusions.

When to Use this Analysis?

Consider using the Kruskal Wallis test in the following cases:

• You have ordinal data.
• Your data follow a nonnormal distribution, and you have a small sample size.
• The median is more relevant to your subject area than the mean.

Learn more about the Normal Distribution .

If you have 3 – 9 groups and more than 15 observations per group or 10 – 12 groups and more than 20 observations per group, you might want to use one-way ANOVA even when you have nonnormal data. The central limit theorem causes the sampling distributions to converge on normality, making ANOVA a suitable choice.

One-way ANOVA has several advantages over the Kruskal Wallis test, including the following:

• More statistical power to detect differences.
• Can handle distributions with different shapes ( Use Welch’s ANOVA ).
• Avoids the interpretation issues discussed above.

In short, use this nonparametric method when you’re specifically interested in the medians, have ordinal data, or can’t use one-way ANOVA because you have a small, nonnormal sample.

Interpreting Kruskal Wallis Test Results

Like one-way ANOVA, the Kruskal Wallis test is an “omnibus” test. Omnibus tests can tell you that not all your groups are equal, but it doesn’t specify which pairs of groups are different.

Specifically, the Kruskal Wallis test evaluates the following hypotheses:

• Null : The average ranks are all the same.
• Alternative : At least one average rank is different.

Again, if the distributions have similar shapes, you can replace “average ranks” with “medians.”

Imagine you’re studying five different diets and their impact on weight loss. The Kruskal Wallis test can confirm that at least two diets have different results. However, it won’t tell you exactly which pairs of diets have statistically significant differences.

So, how do we solve this problem? Enter post hoc tests. Perform these analyses after (i.e., post) an omnibus analysis to identify specific pairs of groups with statistically significant differences. A standard option includes Dunn’s multiple comparisons procedure. Other options include performing a series of pairwise Mann-Whitney U tests with a Bonferroni correction or the lesser-known but potent Conover-Iman method.

Learn about Post Hoc Tests for ANOVA .

Kruskal Wallis Test Example

Imagine you’re a healthcare administrator analyzing the median number of unoccupied beds in three hospitals. Download the CSV dataset: KruskalWallisTest .

For this Kruskal Wallis test, the p-value is 0.029, which is less than the typical significance level of 0.05. Consequently, we can reject the null hypothesis that all groups have the same average rank. At least one group has a different average rank than the others.

Furthermore, if the three hospital distributions have the same shape, we can conclude that the medians differ.

At this point, we might decide to use a post hoc test to compare pairs of hospitals.

Share this:

Reader Interactions

May 20, 2024 at 2:07 pm

Sir kruskal walllis test is Two tailed or one tailed test??

May 20, 2024 at 3:55 pm

It’s a one-tailed test in the same sense that the F-test for one-way ANOVA is one-tailed.

Comments and Questions Cancel reply

Statistics Made Easy

A Guide to Using Post Hoc Tests with ANOVA

An ANOVA is a statistical test that is used to determine whether or not there is a statistically significant difference between the means of three or more independent groups. 

The hypotheses used in an ANOVA are as follows:

The null hypothesis (H 0 ): µ 1  = µ 2  = µ 3 = … = µ k   (the means are equal for each group)

The alternative hypothesis: (Ha): at least one of the means is different from the others

If the p-value  from the ANOVA is less than the significance level, we can reject the null hypothesis and conclude that we have sufficient evidence to say that at least one of the means of the groups is different from the others.

However, this doesn’t tell us  which  groups are different from each other. It simply tells us that not all of the group means are equal.

In order to find out exactly which groups are different from each other, we must conduct a post hoc test  (also known as a multiple comparison test), which will allow us to explore the difference between multiple group means while also controlling for the family-wise error rate.

Technical Note: It’s important to note that we only need to conduct a post hoc test when the p-value for the ANOVA is statistically significant. If the p-value is not statistically significant, this indicates that the means for all of the groups are not different from each other, so there is no need to conduct a post hoc test to find out which groups are different from each other.

The Family-Wise Error Rate

As mentioned before, post hoc tests allow us to test for difference between multiple group means while also controlling for the family-wise error rate . 

In a hypothesis test , there is always a type I error rate, which is defined by our significance level (alpha) and tells us the probability of rejecting a null hypothesis that is actually true. In other words, it’s the probability of getting a “false positive”, i.e. when we claim there is a statistically significant difference among groups, but there actually isn’t.

When we perform one hypothesis test, the type I error rate is equal to the significance level, which is commonly chosen to be 0.01, 0.05, or 0.10. However, when we conduct multiple hypothesis tests at once, the probability of getting a false positive increases.

For example, imagine that we roll a 20-sided dice. The probability that the dice lands on a “1” is just 5%. But if we roll two dice at once, the probability that one of the dice will land on a “1” increases to 9.75%. If we roll five dice at once, the probability increases to 22.6%. 

The more dice we roll, the higher the probability that one of the dice will land on a “1.” Similarly, if we conduct several hypothesis tests at once using a significance level of .05, the probability that we get a false positive increases to beyond just 0.05.

Multiple Comparisons in ANOVA

When we conduct an ANOVA, there are often three or more groups that we are comparing to one another. Thus, when we conduct a post hoc test to explore the difference between the group means, there are several  pairwise  comparisons we want to explore.

For example, suppose we have four groups: A, B, C, and D. This means there are a total of six pairwise comparisons we want to look at with a post hoc test:

A – B (the difference between the group A mean and the group B mean) A – C A – D B – C B – D C – D

If we have more than four groups, the number of pairwise comparisons we will want to look at will only increase even more. The following table illustrates how many pairwise comparisons are associated with each number of groups along with the family-wise error rate:

Notice that the family-wise error rate increases rapidly as the number of groups (and consequently the number of pairwise comparisons) increases. In fact, once we reach six groups, the probability of us getting a false positive is actually above 50%!

This means we would have serious doubts about our results if we were to make this many pairwise comparisons, knowing that our family-wise error rate was so high.

Fortunately, post hoc tests provide us with a way to make multiple comparisons between groups while controlling the family-wise error rate.

Example: One-Way ANOVA with Post Hoc Tests

The following example illustrates how to perform a one-way ANOVA with post hoc tests.

Note: This example uses the programming language R, but you don’t need to know R to understand the results of the test or the big takeaways.

First, we’ll create a dataset that contains four groups (A, B, C, D) with 20 observations per group:

Next, we’ll fit a one-way ANOVA to the dataset:

From the ANOVA table output, we see that the F-statistic is 17.66 and the corresponding p-value is extremely small.

This means we have sufficient evidence to reject the null hypothesis that all of the group means are equal. Next, we can use a post hoc test to find which group means are different from each other.

We will walk through examples of the following post hoc tests:

Tukey’s Test – useful when you want to make every possible pairwise comparison

Holm’s Method – a slightly more conservative test compared to Tukey’s Test

Dunnett’s Correction – useful when you want to compare every group mean to a control mean, and you’re not interested in comparing the treatment means with one another.

Tukey’s Test

We can perform Tukey’s Test for multiple comparisons by using the built-in R function  TukeyHSD()  as follows:

Notice that we specified our confidence level to be 95%, which means we want our family-wise error rate to be .05. R gives us two metrics to compare each pairwise difference:

• Confidence interval for the mean difference (given by the values of  lwr  and  upr )
• Adjusted p-value for the mean difference

Both the confidence interval and the p-value will lead to the same conclusion.

For example, the 95% confidence interval for the mean difference between group C and group A is (0.2813, 1.4309), and since this interval doesn’t contain zero we know that the difference between these two group means is statistically significant. In particular, we know that the difference is positive, since the lower bound of the confidence interval is greater than zero.

Likewise, the p-value for the mean difference between group C and group A is 0.0011, which is less than our significance level of 0.05, so this also indicates that the difference between these two group means is statistically significant.

We can also visualize the 95% confidence intervals that result from the Tukey Test by using the plot() function in R:

If the interval contains zero, then we know that the difference in group means is not statistically significant. In the example above, the differences for B-A and C-B are not statistically significant, but the differences for the other four pairwise comparisons are statistically significant. 

Holm’s Method

Another post hoc test we can perform is holm’s method. This is generally viewed as a more conservative test compared to Tukey’s Test.

We can use the following code in R to perform holm’s method for multiple pairwise comparisons:

This test provides a grid of p-values for each pairwise comparison. For example, the p-value for the difference between the group A and group B mean is 0.20099. 

If you compare the p-values of this test with the p-values from Tukey’s Test, you’ll notice that each of the pairwise comparisons lead to the same conclusion, except for the difference between group C and D. The p-value for this difference was .0505 in Tukey’s Test compared to .02108 in Holm’s Method.

Thus, using Tukey’s Test we concluded that the difference between group C and group D was not statistically significant at the .05 significance level, but using Holm’s Method we concluded that the difference between group C and group D  was  statistically significant. 

In general, the p-values produced by Holm’s Method tend to be lower than those produced by Tukey’s Test.

Dunnett’s Correction

Yet another method we can use for multiple comparisons is Dunett’s Correction. We would use this approach when we want to compare every group mean to a control mean, and we’re not interested in comparing the treatment means with one another.

For example, using the code below we compare the group means of B, C, and D all to that of group A. So, we use group A as our control group and we aren’t interested in the differences between groups B, C, and D. 

From the p-values in the output we can see the following:

• The difference between the group B and group A mean is not statistically significant at a significance level of .05. The p-value for this test is 0.4324 .
• The difference between the group C and group A mean is statistically significant at a significance level of .05. The p-value for this test is 0.0005 .
• The difference between the group D and group A mean is statistically significant at a significance level of .05. The p-value for this test is 0.00004 .

As we stated earlier, this approach treats group A as the “control” group and simply compares every other group mean to that of group A. Notice that there are no tests performed for the differences between groups B, C, and D because we aren’t interested in the differences between those groups.

A Note on Post Hoc Tests & Statistical Power

Post hoc tests do a great job of controlling the family-wise error rate, but the tradeoff is that they reduce the statistical power of the comparisons. This is because the only way to lower the family-wise error rate is to use a lower significance level for all of the individual comparisons.

For example, when we use Tukey’s Test for six pairwise comparisons and we want to maintain a family-wise error rate of .05, we must use a significance level of approximately 0.011 for each individual significance level. The more pairwise comparisons we have, the lower the significance level we must use for each individual significance level.

The problem with this is that lower significance levels correspond to lower statistical power. This means that if a difference between group means actually does exist in the population, a study with lower power is less likely to detect it. 

One way to reduce the effects of this tradeoff is to simply reduce the number of pairwise comparisons we make. For example, in the previous examples we performed six pairwise comparisons for the four different groups. However, depending on the needs of your study, you may only be interested in making a few comparisons.

By making fewer comparisons, you don’t have to lower the statistical power as much.

It’s important to note that you should determine  before  you perform the ANOVA exactly which groups you want to make comparisons between and which post hoc test you will use to make these comparisons. Otherwise, if you simply see which post hoc test produces statistically significant results, that reduces the integrity of the study.

In this post, we learned the following things:

• An ANOVA is used to determine whether or not there is a statistically significant difference between the means of three or more independent groups.
• If an ANOVA produces a p-value that is less than our significance level, we can use post hoc tests to find out which group means differ from one another.
• Post hoc tests allow us to control the family-wise error rate while performing multiple pairwise comparisons.
• The tradeoff of controlling the family-wise error rate is lower statistical power. We can reduce the effects of lower statistical power by making fewer pairwise comparisons.
• You should determine beforehand which groups you’d like to make pairwise comparisons on and which post hoc test you will use to do so.

Featured Posts

Hey there. My name is Zach Bobbitt. I have a Masters of Science degree in Applied Statistics and I’ve worked on machine learning algorithms for professional businesses in both healthcare and retail. I’m passionate about statistics, machine learning, and data visualization and I created Statology to be a resource for both students and teachers alike.  My goal with this site is to help you learn statistics through using simple terms, plenty of real-world examples, and helpful illustrations.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Join the Statology Community

Sign up to receive Statology's exclusive study resource: 100 practice problems with step-by-step solutions. Plus, get our latest insights, tutorials, and data analysis tips straight to your inbox!

By subscribing you accept Statology's Privacy Policy.