[R] Waaaayy off topic...Statistical methods, pub bias, scientific validity

[Mike Marchywka]

> Date: Thu, 6 Jan 2011 23:06:44 -0800
> From: peter.langfelder at gmail.com
> To: r-help at r-project.org
> Subject: Re: [R] Waaaayy off topic...Statistical methods, pub bias, scientific validity
>
> >From a purely statistical and maybe somewhat naive point of view,
> published p-values should be corrected for the multiple testing that
> is effectively happening because of the large number of published
> studies. My experience is also that people will often try several
> statistical methods to get the most significant p-value but neglect to
> share that fact with the audience and/or at least attempt to correct
> the p-values for the selection bias.

You see this everywhere in one form or another from medical to financial
modelling. My solution here is simply to publish more raw data in a computer
readable form, in this case of course something easy to get with R,
so disinterested or adversarial parties can run their own "analysis."
I think there was also a push to create a data base for failed drug
trials that may contain data of some value later. The value of R with
easily available data for a large cross section of users could be to moderate 
problems like the one cited here. 

I almost
slammed a poster here earlier who wanted a simple rule for "when do I use
this test" with something like " when your mom tells you to" since post
hoc you do just about everything to assume you messed up and missed something
but a priori you hope you have designed a good hypothesis. And at the end of
the day, a given p-value is one piece of evidence in the overall objective
of learning about some system, not appeasing a sponsor. Personally I'm a big
fan of post hoc analysis on biotech data in some cases, especially as more pathway or other theory
is published, but it is easy to become deluded if you have a conclusion that you
know JUST HAS TO BE RIGHT. 

Also FWIW, in the few cases I've examined with FDA-sponsor rhetoric, the
data I've been able to get tends to make me side with the FDA and I still hate the
idea of any regulation or access restrictions but it seems to be the only way
to keep sponsors honest to any extent. Your mileage
may vary however, take a look at some rather loud disagreement with FDA
over earlier DNDN panel results, possibly involving threats against critics. LOL.





>
> That being said, it would seem that biomedical sciences do make
> progress, so some of the published results are presumably correct :)
>
> Peter
>
> On Thu, Jan 6, 2011 at 9:13 PM, Spencer Graves
>  wrote:
> >      Part of the phenomenon can be explained by the natural censorship in
> > what is accepted for publication:  Stronger results tend to have less
> > difficulty getting published.  Therefore, given that a result is published,
> > it is evident that the estimated magnitude of the effect is in average
> > larger than it is in reality, just by the fact that weaker results are less
> > likely to be published.  A study of the literature on this subject might
> > yield an interesting and valuable estimate of the magnitude of this
> > selection bias.
> >
> >
> >      A more insidious problem, that may not affect the work of Jonah Lehrer,
> > is political corruption in the way research is funded, with less public and
> > more private funding of research
> > (http://portal.unesco.org/education/en/ev.php-URL_ID=21052&URL_DO=DO_TOPIC&URL_SECTION=201.html).
> >  For example, I've heard claims (which I cannot substantiate right now) that
> > cell phone companies allegedly lobbied successfully to block funding for
> > researchers they thought were likely to document health problems with their
> > products.  Related claims have been made by scientists in the US Food and
> > Drug Administration that certain therapies were approved on political
> > grounds in spite of substantive questions about the validity of the research
> > backing the request for approval (e.g.,
> > www.naturalnews.com/025298_the_FDA_scientists.html).  Some of these
> > accusations of political corruption may be groundless.  However, as private
> > funding replaces tax money for basic science, we must expect an increase in
> > research results that match the needs of the funding agency while degrading
> > the quality of published research.  This produces more research that can not
> > be replicated -- effects that get smaller upon replication.  (My wife and I
> > routinely avoid certain therapies recommended by physicians, because the
> > physicians get much of their information on recent drugs from the
> > pharmaceuticals, who have a vested interest in presenting their products in
> > the most positive light.)
> >