[BioC] Multiple test question in micrarray- FDR

Naomi Altman naomi at stat.psu.edu
Mon Dec 15 01:15:59 CET 2008

The ball model does not apply to microarray studies.  (And the 
probability of drawing the red ball in 20 draws is not 1).

But FDR does apply to microarray studies, and so does a less 
discussed concept, the false nondiscovery rate or FNR.

Suppose I take 20 independent samples of mouse liver tissue - same 
strain, gender ... and hybridize independently to 20 microarrays - 
any platform.
Then arbitrarily divide into 2 groups of size 10.  If there are 
10,000 genes on the array, you should see 1 gene with p-value .0001or 
less, 10 genes with p-value .001 or less, 100 genes with p-value .01 
or less etc.  Now suppose you take the 100 genes with the highest 
degree of differential expression and do a PCR study with independent 
samples.  You should still have 1 gene which is significant with 
p=.01 and 5 genes which are significant at p=.05.

The problem is - there is no systematic difference between the 
samples.  You have detected noise - i.e. chance variation.  If you 
use the same samples to do your PCR, you may get closer to 100% 
"significance" for the selected genes, because the variation that 
caused the false detection will still be in the sample unless it was 
due only to the hybridization.

FDR is an estimate of the excess of significant findings, compared to 
what is expected by chance.  You can reduce FDR greatly by doing 
independent follow-up studies (on another microarray or on another 
platform such as PCR).  You cannot reduce FDR much by reusing the 
same samples on a different platform, although you will reduce 
affects due to technical variation.

However, FDR reduces your power to detect differential 
expression.  This means that you will have higher FNR if you use 
multiple comparisons adjustments.  Again, if you do independent 
follow-up studies, you can reduce FNR.

The purpose of the FDR computation is to reduce effort wasted on 
large gene lists which are mostly reporting noise.  But if your 
genelist is smaller than you think is reasonable, you may certainly 
follow up a larger set of genes and sorting by p-value will give you 
the most reasonable set of genes to follow up.  Again,
the only valid follow-up uses independent samples and independent platforms.  \

At 02:38 PM 12/14/2008, Wayne Xu wrote:
>Dear Naomi,
>I may have a silly question. I read a few papers on microarray 
>multiple test, I understood what points they were trying to make. 
>But I still have doubts about it. Since now many journal reviewers 
>require the FDR for microarray differential expresses genes in 
>manuscripts, I really want to clear my doubts.
>1). The mathematics model is different from the biology model:
>The typical math model to bring up the multiple test issue is 
>following example: 20 balls in a box with 1 in red and 19 in blue. 
>The possibility of picking up the red ball from the box each time is 
>1/20, i.e 0.05. If draw 20 times, the chance is 0.05 multiplied by 20 is 1.
>Suppose the red represents false positive, if draw one time the FDR 
>is 0.05, if 20 times then FDR is 1. People bring this multiple test 
>issue into microarray data analysis. But in microarray, at least two 
>aspects are different from this math model:
>a). The raw P values are determined by the expression values of 
>samples, not affected by the total number of genes.  So it is 
>different from above example of 1 out of 20 is 0.05.
>b). Pick up a ball and then put it back to the box, you have chance 
>to pick up the exactly same ball twice or more. But in microarray, 
>each genes are tested individually at the same time, and each gene 
>only tested exactly once.
>They are obviously different. If this math model is the only reason 
>that brought up the multiple test issue in microarray, it may be a 
>misleading (I may be silly, since no one else doubts about multiple 
>test in microarray?)
>2). Not make biological sense:
>Suppose a gene called XYZ has a raw P value of 0.00001 in two group 
>T test, and it was validated by biological test, e.g. RT-PCR. If the 
>micoarray chip has 40,000 genes, then by whatever adjustment  FDR 
>method, the adj P-value may be 0.4 or lower or higher. If I use FDR 
>cutoff 0.1, this XYZ gene has higher FDR and is not in my interest 
>positive gene list.
>OK, now I play a math game, filter gene by variance or other, shrink 
>the gene list to 5000 (since XYZ gene has low P value, suppose it is 
>within the 5000). Then the XYZ has low FDR and in my interest 
>differential gene list. But this is just a math game!
>The biological reality is XYZ is positive, this positive is 
>determined by, for example 4 control samples and 4 treatment 
>samples, the mean may be big different, and within group variance is 
>very small. and RT-PCR validated. This reality can not be changed by 
>whatever number of genes to be tested. The raw P value is close the 
>biological reality, and it is good to represent the biological 
>reality. The multiple test here just make you feel happier but not a 
>biological sense.
>FDR is a very useful term in many biological cases.  But it seems 
>not a good example here for microarray?
>Please help to clear it up.
>Thank you,
>Naomi Altman wrote:
>>Remember that FDR is a rate - i.e. the expected false discovery rate.
>>If the set of genes is changeds, FDR will change because the 
>>comparison set is different.  This is NOT the same as a p-value, 
>>which depends only on the value of the current test statistic.
>>The same thing happens with FWER, because these methods control the 
>>probability of making at least one mistake, which clearly depends 
>>on which set of tests are performed.
>>At 03:11 PM 12/13/2008, Sean Davis wrote:
>>>On Sat, Dec 13, 2008 at 12:36 PM, Wayne Xu <wxu at msi.umn.edu> wrote:
>>> > Hello,
>>> > I am not sure this is a right place to ask this question, but it is about
>>> > micrarray data analysis:
>>> >
>>> > In two group t test, the multiple test Q values are depending 
>>> on the total
>>> > number of genes in the test. If I filter the gene list first, 
>>> for example, I
>>> > only use those genes that have1.2 fold changes for T test and 
>>> multiple test,
>>> > this gene list is much smaller than the total gene list, then 
>>> the multiple
>>> > test q values are much smaller.
>>> >
>>> > Do you think above is a correct way? People who do not do that way may
>>> > consider the statistical power may be lost? But how much power 
>>> lost and how
>>> > to calculate the power in this case?
>>>No, you cannot filter based on fold change.  However, you can filter
>>>based on variance or some other measure that does not depend on the
>>>two groups being compared.  Anything that filters genes based on
>>>"knowing" the two groups will lead to a biased test.  Remember that
>>>filtering removes genes from consideration from further analysis.
>>>For further details, there are MANY discussions of this topic in the
>>>mailing list.
>>> > When people report multiple test Q values, they usually do not 
>>> mention how
>>> > many genes are used in this multiple test. You can get different Q values
>>> > (even use the same method, e.g. Benjamin and Holm adjust 
>>> method) in the same
>>> > dataset. Then how can it make sense if the same genes have different Q
>>> > values?
>>>A good manuscript should describe in detail the preprocessing and
>>>filtering steps, the statistical tests used, and the methods for
>>>correcting for multiple testing.  You are correct that many papers do
>>>not do so.
>>>As for different q-values in the same dataset using different methods,
>>>it is important to note that one should not do an analysis, get a
>>>result, and then, based on that result, go back and redo the analysis
>>>with different parameters to get a "better" result.  It is very
>>>important that each step of an analysis (preprocessing, filtering,
>>>testing, multiple-testing correction) be justifiable independent of
>>>the other steps in order for the results to be interpretable.
>>>Bioconductor mailing list
>>>Bioconductor at stat.math.ethz.ch
>>>Search the archives: 
>>Naomi S. Altman                                814-865-3791 (voice)
>>Associate Professor
>>Dept. of Statistics                              814-863-7114 (fax)
>>Penn State University                         814-865-1348 (Statistics)
>>University Park, PA 16802-2111
>Bioconductor mailing list
>Bioconductor at stat.math.ethz.ch
>Search the archives: 

Naomi S. Altman                                814-865-3791 (voice)
Associate Professor
Dept. of Statistics                              814-863-7114 (fax)
Penn State University                         814-865-1348 (Statistics)
University Park, PA 16802-2111

More information about the Bioconductor mailing list