[R] prop.test CI depends on null hypothesis?

[Aaron Rendahl]

I believe this is correct behavior for computing the p-value, though the
wording is awkward in that it implies that R is not implementing the
continuity correction in this situation, when in fact, this behavior is
part of how the continuity correction is defined. The correction simply
treats the normal approximation as appropriately discrete, so (to translate
to a binomial variable) computes P(X > 12) using P(X > 11.5). The case the
documentation discusses is simply the case where the null hypothesis falls
within the discrete band corresponding to the observed value; in this only
enough of the correction is used so that the test statistic is
appropriately zero and the p-value is 1.

However, this is not correct behavior for the confidence interval. There is
nothing in any of the listed documentation that would support such
behavior, and additionally, it doesn't make sense for a confidence interval
to depend on a null parameter. If continuity correction is desired, the
edges of the confidence bound should still be fully adjusted even when the
observed proportion is close to the null parameter. What's currently
happening is that it's not adjusted at all when the observed proportion
equals the null proportion, and in cases where it is not equal but still
close enough that the correction is adjusted, the confidence intervals are
neither "with" correction" or "without" correction but instead somewhere in
between!

An additional confusing matter is how R reports whether the test was
performed "with" or "without" continuity correction; this is determined in
code by whether or not the adjusted correction is zero or not. This happens
when the observed proportion equals the null proportion, so whenever that
happens, it's reported "without" continuity correction, so this "flips" on
the user in this case. Though oddly (to the user), changing the null p by a
tiny amount gives only tiny changes to the result but then it is reported
"with" correction.

This behavior has presumably been in R for a long time (though I haven't
checked the code history), so I would love to have feedback from the R-help
community about:
 * does the current behavior really make sense, and I've just misunderstood
something?
 * is there documentation or discussions about this behavior out there
somewhere that I've missed?
 * if this really is a "new" discovery, how best to bring it to the
attention of those who can decide what to do about it?

Thanks!

On Mon, Oct 21, 2019 at 11:33 AM Aaron Rendahl <rend0020 using umn.edu> wrote:

> Why does prop.test use continuity correction "only if it does not exceed
> the difference between sample and null proportions in absolute value"?  I'm
> referring here to the single group method, though I believe there is a
> similar issue with the two group method.
>
> What this means in practice is that the confidence interval changes
> depending on the null hypothesis; see examples below. This is unexpected,
> and I have been unable to find any documentation explaining why this is
> done (see links below examples).
>
> ## when the null proportion is equal to the sample proportion, it does not
> ## use the continuity correction, even when one is asked for
>
>   > prop.test(30,60,p=0.5, correct=TRUE)
>
> 1-sample proportions test without continuity correction
>
> data:  30 out of 60, null probability 0.5
> X-squared = 0, df = 1, p-value = 1
> alternative hypothesis: true p is not equal to 0.5
> 95 percent confidence interval:
>  0.3773502 0.6226498
> sample estimates:
>   p
> 0.5
>
> ## however, when the null proportion is not equal to the sample
> proportion,
> ## it does use the continuity correction when it is asked for.
>
> > prop.test(30,60,p=0.499, correct=TRUE)
>
> 1-sample proportions test with continuity correction
>
> data:  30 out of 60, null probability 0.499
> X-squared = 0, df = 1, p-value = 1
> alternative hypothesis: true p is not equal to 0.499
> 95 percent confidence interval:
>  0.3764106 0.6235894
> sample estimates:
>   p
> 0.5
>
>
> The documentation refers to Newcombe's 1998 Statistics in Medicine
> article; I read through this and found nothing about not using the
> continuity correction in this situation.
>
> https://doi.org/10.1002/(SICI)1097-0258(19980430)17:8%3C857::AID-SIM777%3E3.0.CO;2-E
>
> On this mailing list, there was a 2013 post "prop.test correct true and
> false gives same answer", which was answered only with the quote from the
> help page: https://stat.ethz.ch/pipermail/r-help/2013-March/350386.html
>
> I also found several questions asking which Newcombe method is
> implemented, which didn't elicit specific answers; here's one from 2011:
> https://stat.ethz.ch/pipermail/r-help/2011-April/274086.html
>

-- 
Aaron Rendahl, Ph.D.
Assistant Professor of Statistics and Informatics
College of Veterinary Medicine, University of Minnesota
295L AS/VM, 612-301-2161

	[[alternative HTML version deleted]]