testing

A statistical hypothesis test involving rearranging labels on data points to generate a null distribution. This technique is particularly useful when distributional assumptions are questionable or when conventional parametric tests are inappropriate. As an example, consider two groups where a researcher aims to assess whether they originate from the same population. The procedure involves pooling the data from both groups, then repeatedly randomly assigning each data point to either group A or group B, thus creating simulated datasets assuming no true difference between the groups. For each simulated dataset, a test statistic (e.g., the difference in means) is calculated. The observed test statistic from the original data is then compared to the distribution of the simulated test statistics to obtain a p-value.

This approach offers several advantages. Its non-parametric nature renders it robust against departures from normality or homoscedasticity. Its also well-suited for small sample sizes where parametric assumptions are difficult to verify. The method can be traced back to early work by Fisher and Pitman, predating the availability of widespread computational power. The increased availability of computing resources has vastly improved its practicality, allowing for thorough exploration of the null distribution and thereby enhancing the validity of inferences.

Read more