ANOVA Post-hoc tests

Dr. Peng Zhao (✉ peng.zhao@xjtlu.edu.cn)

Department of Health and Environmental Sciences
Xi’an Jiaotong-Liverpool University

1 Learning objectives

  1. What is a post-hoc test and why we need it.
  2. Carry out step-by-step post-hoc tests, including the Fisher’s Lease Significant Difference test and the Bonferroni t-test, for ANOVA.

2 Post-hoc tests

Post-hoc

  • Latin, “after this”
  • Applied only after the ANOVA that yields a significant difference.

Example: Rats on diets

A biologist studies the weight gain of male lab rats on diets over a 4-week period. Three different diets are applied.

dtf <- data.frame(diet1 = c(90, 95, 100),
                  diet2 = c(120, 125, 130),
                  diet3 = c(125, 130, 135))
dtf2 <- stack(dtf)
names(dtf2) <- c("wg", "diet")
wg_aov <- aov(wg ~ diet, data = dtf2)
summary(wg_aov)

Which group(s) is/are different from others? Post-hoc test.

Visually:

library(ggplot2)
ggplot(dtf2) + geom_boxplot(aes(wg, diet))

3 Fisher’s Least Significant Difference (LSD) Test

3.1 Principle

Pair-wise comparisons of all the groups based on the t-test.

\[ L S D=t_{\alpha / 2} \sqrt{S_{p}^{2}\left(\frac{1}{n_1}+\frac{1}{n_2}+\cdots\right)} \]

\[ S_{p}^{2}=\frac{\left(n_{1}-1\right) S_{1}^{2}+\left(n_{2}-1\right) S_{2}^{2}+\left(n_{3}-1\right) S_{3}^{2}+\cdots}{\left(n_{1}-1\right)+\left(n_{2}-1\right)+\left(n_{3}-1\right)+\cdots} \]

  • \(S_{p}^{2}:\) (pooled standard deviation; some use Mean Standard Error)
  • \(t_{\alpha / 2}: \mathrm{t}\) critical value at \(\alpha=0.025\)
  • Degree of freedom: \(N - k\)
    • \(N\): total observations
    • \(k\): number of factors

Usage

  • If \(\left|\bar{x}_{1}-\bar{x}_{2}\right|>L S D\), then the difference of \(x_1\) group and \(x_2\) group is significant at \(\alpha\).
  • In multiple comparisons (\(k\) factors), the number of comparison needed is: \(\frac{k(k-1)}{2}\)

3.2 Example

library(agricolae)
LSD.test(wg_aov, "diet", p.adj = "bonferroni")

Conclusion: At \(\alpha = 0.05\), Diet 2 and Diet 3 are significantly different from Diet 1 in the mean weight gain, while Diet 2 is not significantly different from Diet 3.

4 Bonferroni t-test

4.1 Principles

A multiple-comparison post-hoc test, which sets the significance cut off at \(\alpha/m\) for each comparison, where \(m\) represents the number of comparisons we apply.

Overall chance of making a Type I error:

m <- 1:100
siglevel <- 0.05
1 - (1 - (siglevel / m)) ^ m

4.2 Example

Example: Rats on diets

diet_pt <- pairwise.t.test(dtf2$wg, dtf2$diet, pool.sd = FALSE, var.equal = TRUE, p.adj = "none")
diet_pt$p.value < 0.05/m 

pairwise.t.test(dtf2$wg, dtf2$diet, pool.sd = FALSE,var.equal = TRUE, p.adj = "bonferroni")
diet_pt$p.value < 0.05

Conclusion: the same as the LSD test.

5 Further readings