Permutation Hypothesis Testing

Tutorial about its theory:

Computational implementation of the theory:

 

Summary

1. Specify $H_0$ (null hypothesis) and $H_1$. For example, $H_0$ : weight gain is irrelevant to a diet.
2. Choose a test statistic: (absolute) difference in mean or median.
3. Compute a distribution of the test statistic using the permutations of the observed data.
4. Compute $p-value$ which is the probability of getting the permutation test statistic same as the observed test statistic or more extreme if the null hypothesis is true.
5. If $p-value$ is smaller than the significance level $\alpha$, we can reject the null hypothesis $H_0$.

Note that the test statistic would be close to zero if the weight gain were irrelevant to two groups. If $H_0$ is true, $p-value$ should be relatively large, and if $H_0$ is false, $p-value$ should be fairly small (if it's smaller than $\alpha$, we can reject $H_0$).