If the test statistic is continuous, it will reach the significance level exactly.
Hence, when a result of statistical analysis is termed an “exact test” or specifies an “exact p-value”, this implies that the test is defined without parametric assumptions and is evaluated without making use of approximate algorithms.
In principle, however, this could also signify that a parametric test has been employed in a situation where all parametric assumptions are fully met, but it is in most cases impossible to prove this completely in a real-world situation.
The basic equation underlying exact tests is where: and where the sum ranges over all outcomes y (including the observed one) that have the same value of the test statistic obtained for the observed sample x, or a larger one.
Suppose Pearson's chi-squared test is used to ascertain whether a six-sided die is "fair", indicating that it renders each of the six possible outcomes equally often.
The test statistic is where Xk is the number of times outcome k is observed.
If the null hypothesis of "fairness" is true, then the probability distribution of the test statistic can be made as close as desired to the chi-squared distribution with 5 degrees of freedom by making the sample size n sufficiently large.
On the other hand, if n is small, then the probabilities based on chi-squared distributions may not be sufficiently close approximations.