Estimated P-values in Discrete Models: Asymptotic and non-asymptotic effects

Chris Lloyd, Melbourne Business School

Abstract

The exact null distribution of a P-value typically depends on nuisance parameters unspecified under the null. For discrete models and standard approximate P-values, this dependence can be quite strong. The estimated (or bootstrap) P-value is the exact probability of the P-value being no larger than its observed value, with the null estimate of the nuisance parameter substituted. For continuous models, it is known that such `bootstrap' P-values deviate from uniformity by terms of order m^{-3/2}, where m is a measure of sample size. The main difficulty with discrete models is the breakdown of asymptotics near the boundary. The aim of this paper is to numerically examine the accuracy of standard and bootstrap P-values for discrete models. We examine a range of binomial models, test statistics and look at testing both canonical and non-canonical parameters.

When departures from uniformity are averaged across the nuisance parameter, we find that errors of bootstrap P-values appear to improve at a rate of order m^{-1}. When interest lies in the maximum error, we find that errors hardly seem to decrease with sample size. Nevertheless, bootstrap P-values enjoy accuracy an order of magnitude better than standard P-values, even for small sample sizes. The reasons why bootstrap works so well in this regard has nothing to do with asymptotics and it is explained how bootstrap can automatically correct small departures from the likelihood based ordering of the sample space.