[Rd] Problem with R >3.0.0
Peter Langfelder
peter.langfelder at gmail.com
Tue Aug 20 19:53:02 CEST 2013
Hi Samuel,
WGCNA currently does not support calculations with matrices larger
than the old R limit, and it will take some time to update it to
support the large matrices. Furthermore, WGCNA relies on other
functions (most notably hclust) that would have to be updated as well
to support long vectors.
In the meantime I suggest using the "blockwise" functions to handle
large data sets, or, if possible, reducing the number of genes to less
than the old limit of 46340 or so.
Sorry I can't be of more help.
Best,
Peter
On Tue, Aug 20, 2013 at 10:42 AM, Shelton, Samuel
<SheltonS at stemcell.ucsf.edu> wrote:
> Hi all,
>
> Thanks for getting back to me. We would like to move over to v3.0.0 on our
> cluster so that we can build matrices larger than 46300*46300 (limit in R
> <3.0.0)
> but so far we can't get things to work with R v3.0.0 and higher. I am
> trying to trouble shoot at the moment and I am now thinking that the
> problem is actually with the diag function that has been rewritten in
> version 3.0.0.
>
>
> The problem is definitely with the diag function and it does not occur on
> smaller matrices (20000*20000) and I think it maybe a bug.
> This illustrates the problem:
>
> This was done on an iMac i5 with OSX 10.8.5 16GB Ram and with R 3.0.1 (but
> I do see the same for 3.0.0). This does not occur when I run it with R
> 2.15.2.
>
>
> mat1=matrix(rexp(20000^2), 20000)
>
> mat1[1:10,1:10]
> [,1] [,2] [,3] [,4] [,5] [,6]
> [,7] [,8] [,9] [,10]
> [1,] 0.1829090 0.39867734 0.80499126 4.1746377 0.20717066 1.1477365
> 0.469843567 2.57767543 0.17449595 0.01949358
> [2,] 0.5731522 0.15835939 0.29165029 0.6781249 0.64553728 2.4438404
> 2.140374938 0.40091195 0.51201369 0.98904490
> [3,] 0.3250310 0.09934147 0.79962549 0.4933385 0.30473422 0.4556765
> 0.002640034 0.90606861 2.58772944 0.89884208
> [4,] 1.4195017 0.16082660 0.01377838 0.2115803 1.43992672 0.3883675
> 0.040903805 0.51011305 0.41998024 0.44209926
> [5,] 0.8328441 1.10335604 0.11875332 0.1600287 0.17333324 0.3388678
> 3.206179119 0.52170966 1.03084845 0.05843232
> [6,] 1.3179906 0.76376188 1.24231798 0.9424030 0.04440514 1.0237664
> 2.547528816 1.35629450 0.87983354 0.25236343
> [7,] 0.6990544 1.17003075 0.66063936 0.8632534 0.28965611 0.6718020
> 1.137348735 0.08371053 0.23144290 0.18915132
> [8,] 0.9908026 1.20471979 0.08816010 0.2652131 0.03537790 0.3295816
> 0.144371435 3.03299285 0.09728111 0.39890260
> [9,] 0.9557305 0.29196500 0.43955758 0.7332643 2.03457020 0.5858431
> 2.437192399 0.34689557 0.02039205 0.54898488
> [10,] 3.7220703 0.13572389 0.18888673 0.5683698 1.79209016 1.3495723
> 0.571159401 0.63375850 0.63221987 1.32840290
>> mat1[19990:20000,19990:20000]
> [,1] [,2] [,3] [,4] [,5] [,6]
> [,7] [,8] [,9] [,10] [,11]
> [1,] 1.7910910 0.5719982 0.38689588 1.2157545685 0.8530179 1.464105574
> 0.5986705 1.1623393 0.55244563 0.1770146 0.4326310
> [2,] 0.2862914 1.5267870 0.98214645 0.0004617244 0.6395319 0.075217874
> 0.6725620 0.2403549 0.08436217 0.1435451 0.7487862
> [3,] 2.0492301 0.7216115 0.16951284 0.2726676762 2.1893806 1.202518385
> 0.9897710 1.4813026 2.42517705 0.3398811 0.7285074
> [4,] 0.6538994 0.2437594 2.08848881 0.3917212249 0.4441824 0.433749415
> 1.3022991 1.3695241 0.07057642 0.4296937 2.9307556
> [5,] 2.3688094 2.3970048 0.03545232 0.5986997508 0.8914097 0.497023176
> 0.4210650 1.5337767 0.01141066 1.1562830 1.0572076
> [6,] 2.0626934 0.6186995 0.99197835 1.4794321654 0.1549314 1.296227000
> 0.2790942 0.9327613 0.84131377 0.8782590 0.3279970
> [7,] 1.2423823 0.2385994 0.11390071 2.0745023842 1.9152523 0.754186281
> 1.5474078 2.5899490 5.19298969 1.4680934 1.0537164
> [8,] 1.3657070 1.9502828 1.07681438 0.9339731540 1.7532474 0.186193421
> 1.8699504 1.9187339 5.13248671 0.4621520 0.4753582
> [9,] 0.6512000 0.5104660 0.17820166 0.3965162944 0.0919119 0.187808660
> 0.7391137 0.1574844 0.65985494 0.4066742 0.8072494
> [10,] 0.7435028 1.1395666 2.46096009 0.7060164691 1.7965986 0.008278685
> 0.4642319 0.1582297 1.71676326 0.3662139 0.7864957
> [11,] 0.3537041 0.6622001 2.01642141 1.8225423060 0.3295436 1.260737179
> 0.8430396 0.5132811 0.30547431 1.6088725 0.4001791
>
> diag(mat1)=0
>
> mat1[1:10,1:10]
> [,1] [,2] [,3] [,4] [,5] [,6]
> [,7] [,8] [,9] [,10]
> [1,] 0.0000000 0.39867734 0.80499126 4.1746377 0.20717066 1.1477365
> 0.469843567 2.57767543 0.17449595 0.01949358
> [2,] 0.5731522 0.00000000 0.29165029 0.6781249 0.64553728 2.4438404
> 2.140374938 0.40091195 0.51201369 0.98904490
> [3,] 0.3250310 0.09934147 0.00000000 0.4933385 0.30473422 0.4556765
> 0.002640034 0.90606861 2.58772944 0.89884208
> [4,] 1.4195017 0.16082660 0.01377838 0.0000000 1.43992672 0.3883675
> 0.040903805 0.51011305 0.41998024 0.44209926
> [5,] 0.8328441 1.10335604 0.11875332 0.1600287 0.00000000 0.3388678
> 3.206179119 0.52170966 1.03084845 0.05843232
> [6,] 1.3179906 0.76376188 1.24231798 0.9424030 0.04440514 0.0000000
> 2.547528816 1.35629450 0.87983354 0.25236343
> [7,] 0.6990544 1.17003075 0.66063936 0.8632534 0.28965611 0.6718020
> 0.000000000 0.08371053 0.23144290 0.18915132
> [8,] 0.9908026 1.20471979 0.08816010 0.2652131 0.03537790 0.3295816
> 0.144371435 0.00000000 0.09728111 0.39890260
> [9,] 0.9557305 0.29196500 0.43955758 0.7332643 2.03457020 0.5858431
> 2.437192399 0.34689557 0.00000000 0.54898488
> [10,] 3.7220703 0.13572389 0.18888673 0.5683698 1.79209016 1.3495723
> 0.571159401 0.63375850 0.63221987 0.00000000
>
> mat1[19990:20000,19990:20000]
> [,1] [,2] [,3] [,4] [,5] [,6]
> [,7] [,8] [,9] [,10] [,11]
> [1,] 0.0000000 0.5719982 0.38689588 1.2157545685 0.8530179 1.464105574
> 0.5986705 1.1623393 0.55244563 0.1770146 0.4326310
> [2,] 0.2862914 0.0000000 0.98214645 0.0004617244 0.6395319 0.075217874
> 0.6725620 0.2403549 0.08436217 0.1435451 0.7487862
> [3,] 2.0492301 0.7216115 0.00000000 0.2726676762 2.1893806 1.202518385
> 0.9897710 1.4813026 2.42517705 0.3398811 0.7285074
> [4,] 0.6538994 0.2437594 2.08848881 0.0000000000 0.4441824 0.433749415
> 1.3022991 1.3695241 0.07057642 0.4296937 2.9307556
> [5,] 2.3688094 2.3970048 0.03545232 0.5986997508 0.0000000 0.497023176
> 0.4210650 1.5337767 0.01141066 1.1562830 1.0572076
> [6,] 2.0626934 0.6186995 0.99197835 1.4794321654 0.1549314 0.000000000
> 0.2790942 0.9327613 0.84131377 0.8782590 0.3279970
> [7,] 1.2423823 0.2385994 0.11390071 2.0745023842 1.9152523 0.754186281
> 0.0000000 2.5899490 5.19298969 1.4680934 1.0537164
> [8,] 1.3657070 1.9502828 1.07681438 0.9339731540 1.7532474 0.186193421
> 1.8699504 0.0000000 5.13248671 0.4621520 0.4753582
> [9,] 0.6512000 0.5104660 0.17820166 0.3965162944 0.0919119 0.187808660
> 0.7391137 0.1574844 0.00000000 0.4066742 0.8072494
> [10,] 0.7435028 1.1395666 2.46096009 0.7060164691 1.7965986 0.008278685
> 0.4642319 0.1582297 1.71676326 0.0000000 0.7864957
> [11,] 0.3537041 0.6622001 2.01642141 1.8225423060 0.3295436 1.260737179
> 0.8430396 0.5132811 0.30547431 1.6088725 0.0000000
>
> mat1=matrix(rexp(38000^2), 38000)
> dim(mat1)
> [1] 38000 38000
> mat1[1:10,1:10]
> [,1] [,2] [,3] [,4] [,5] [,6]
> [,7] [,8] [,9] [,10]
> [1,] 3.8622815 0.1357886 1.64090976 0.4494637 0.812315613 0.5328906
> 0.45672475 0.2891504 1.57087882 1.27375802
> [2,] 3.8229940 0.1540735 0.30189392 1.2100152 0.003323061 0.7195875
> 0.60251052 1.9820380 0.18637086 0.05154236
> [3,] 2.0411498 0.8371707 0.02714550 1.3032572 0.330472063 0.3502267
> 0.11908140 0.4155857 3.46471729 0.31890778
> [4,] 1.5503390 0.7377494 2.01433675 0.6109255 1.844484309 1.2492693
> 0.09365743 0.4006219 2.37769616 0.38643521
> [5,] 0.4815804 0.5824312 0.61003728 0.4782871 0.526454982 0.1207842
> 0.93567987 1.7369767 4.47922786 0.20033928
> [6,] 0.3791645 0.1015489 1.96832962 1.5417178 1.030250434 0.1362716
> 1.72807083 0.2570055 0.02127689 0.80225716
> [7,] 1.5212795 2.8133952 0.15990367 0.4337506 0.526532536 2.9926685
> 0.01432572 0.6064162 0.69264596 0.50871566
> [8,] 1.2600365 0.1901277 2.34806048 1.1472887 0.141571521 2.0355007
> 1.12583466 0.3391067 0.18707165 3.71877247
> [9,] 3.0197258 2.3693633 0.94571337 0.2756933 0.938999190 1.5892456
> 0.18612994 1.0498866 1.89162156 1.56643880
> [10,] 0.3573243 0.3047595 0.01894034 2.4666841 0.660994174 0.2248711
> 0.25436398 1.1275389 0.20960212 0.63957112
>
>
>
>
> mat1[37990:38000,37990:38000]
> [,1] [,2] [,3] [,4] [,5] [,6]
> [,7] [,8] [,9] [,10] [,11]
> [1,] 0.8349326 1.32098263 2.68879316 0.52361090 1.4066094 0.00754308
> 2.4489865 1.76284621 0.09097126 2.1598758 0.8805701
> [2,] 2.5754994 1.15753606 1.76895066 3.06645700 0.8767225 0.33247639
> 1.5726808 0.12698495 0.75271682 6.4336476 0.1330457
> [3,] 2.5737957 0.58234929 0.40403205 0.34882433 0.4074048 1.54867135
> 2.5971068 0.27276140 0.56926494 0.2129180 1.3027215
> [4,] 0.9352113 0.46839288 0.41284388 1.40216119 0.8936151 2.98304058
> 0.4350446 1.14864094 0.26756970 2.6662998 0.1802141
> [5,] 1.6430435 1.22137017 0.06943644 0.03251737 0.2142083 0.16964865
> 1.6099314 0.95709429 0.22884734 0.5087986 0.9048555
> [6,] 2.5340159 1.28618183 3.86698388 1.05946189 3.5006776 0.39320471
> 0.4927357 0.66159842 0.18235678 0.5172896 2.2221550
> [7,] 0.1961937 0.17305031 0.62327325 3.14622730 3.4905449 3.08823676
> 0.5282165 0.48879156 0.11807913 0.7372706 0.9975103
> [8,] 0.8392414 0.17107463 1.08732839 0.28981611 0.4722655 0.17587788
> 1.7426814 0.13985161 1.86885446 1.9031580 1.1088431
> [9,] 3.4648342 0.33834943 0.07891645 0.07206860 1.1792628 0.54092203
> 0.8141844 2.82687085 0.78395229 0.5313417 3.2164664
> [10,] 0.1524299 0.02045616 0.67881610 1.51491647 2.4390115 0.89033581
> 0.9026651 0.46858464 1.47711638 2.1821178 2.2149071
> [11,] 0.9397162 0.42376104 0.20034405 1.47672836 1.0461904 0.97296202
> 2.1658717 0.04487329 0.30611082 1.1680312 0.9952517
>
> diag(mat1)=0
>
> mat1[1:10,1:10]
> [,1] [,2] [,3] [,4] [,5] [,6]
> [,7] [,8] [,9] [,10]
> [1,] 0.0000000 0.1357886 1.64090976 0.4494637 0.812315613 0.5328906
> 0.45672475 0.2891504 1.57087882 1.27375802
> [2,] 3.8229940 0.0000000 0.30189392 1.2100152 0.003323061 0.7195875
> 0.60251052 1.9820380 0.18637086 0.05154236
> [3,] 2.0411498 0.8371707 0.00000000 1.3032572 0.330472063 0.3502267
> 0.11908140 0.4155857 3.46471729 0.31890778
> [4,] 1.5503390 0.7377494 2.01433675 0.0000000 1.844484309 1.2492693
> 0.09365743 0.4006219 2.37769616 0.38643521
> [5,] 0.4815804 0.5824312 0.61003728 0.4782871 0.000000000 0.1207842
> 0.93567987 1.7369767 4.47922786 0.20033928
> [6,] 0.3791645 0.1015489 1.96832962 1.5417178 1.030250434 0.0000000
> 1.72807083 0.2570055 0.02127689 0.80225716
> [7,] 1.5212795 2.8133952 0.15990367 0.4337506 0.526532536 2.9926685
> 0.00000000 0.6064162 0.69264596 0.50871566
> [8,] 1.2600365 0.1901277 2.34806048 1.1472887 0.141571521 2.0355007
> 1.12583466 0.0000000 0.18707165 3.71877247
> [9,] 3.0197258 2.3693633 0.94571337 0.2756933 0.938999190 1.5892456
> 0.18612994 1.0498866 0.00000000 1.56643880
> [10,] 0.3573243 0.3047595 0.01894034 2.4666841 0.660994174 0.2248711
> 0.25436398 1.1275389 0.20960212 0.00000000
>
>
> ## After calling the diag function the bottom of the matrix is all set to
> 0.
>
>
> mat1[37990:38000,37990:38000]
> [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11]
> [1,] 0 0 0 0 0 0 0 0 0 0 0
> [2,] 0 0 0 0 0 0 0 0 0 0 0
> [3,] 0 0 0 0 0 0 0 0 0 0 0
> [4,] 0 0 0 0 0 0 0 0 0 0 0
> [5,] 0 0 0 0 0 0 0 0 0 0 0
> [6,] 0 0 0 0 0 0 0 0 0 0 0
> [7,] 0 0 0 0 0 0 0 0 0 0 0
> [8,] 0 0 0 0 0 0 0 0 0 0 0
> [9,] 0 0 0 0 0 0 0 0 0 0 0
> [10,] 0 0 0 0 0 0 0 0 0 0 0
> [11,] 0 0 0 0 0 0 0 0 0 0 0
>
> It looks like there is an issue with larger matrices when calling diag
> function and it has nothing to do with WGCNA.
>
>
>
>
>
>
> On 8/20/13 9:43 AM, "Peter Langfelder" <peter.langfelder at gmail.com> wrote:
>
>>Hi Sam,
>>
>>I assume you mean that correlation for _genes_ (not samples)
>>11262:30000 is 0? I am the maintainer of the WGCNA package but
>>unfortunately I don't have access to a Mac big enough to try
>>30000x30000 correlation matrix, but I would be thankful if you could
>>try reproducing the problem with smaller matrices (e.g. 20000x20000)
>>and try to produce a small reproducible example by
>>generating the data using say rnorm, say like this:
>>
>>nGenes = 20000 # as small as possible that still produces the error
>>nSamples = 100
>>datExpr1 = matrix(rnorm(nSamples * nGenes), nSamples, nGenes)
>>
>>simMat = bicor(datExpr1, use = 'p')
>>
>>Best,
>>
>>Peter
>>
>>On Mon, Aug 19, 2013 at 11:53 AM, Shelton, Samuel
>><SheltonS at stemcell.ucsf.edu> wrote:
>>> Dear R developer,
>>>
>>> I am an R user and am currently having a problem with versions of R
>>>>3.0.0. We build larger correlation matrices of >30000 with pairwise
>>>>correlations made. We have been using a line of code
>>>
>>> simMat=bicor(datExpr1,use="p")
>>>
>>> To build the similarity matrix with datExpr1 being a matrix with genes
>>>as columns (>30000) and rows being samples. This code works fine on an
>>>iMac running OSX 10.8.5 and using R 2.15.2 and the Rblas library. When I
>>>try to run the same code with R version 3.0.0 and 3.0.1 it only
>>>partially builds the matrix. It will return correlations for samples
>>>1:11262 but from 11262:30000 the matrix is full of 0's.
>>>
>>> We would like to be able to use R 3.0.0 to allow us to build larger
>>>correlation matrices on our cluster but we can't use it at the moment
>>>due to this problem. My colleague has previously had a similar issue
>>>when using snow leopard and this was to do with a compatibility issue
>>>between snow leopard and the veclib. Has something changed in version
>>>3.0.0 to give an incompatibility with the veclib?
>>>
>>> Many thanks
>>>
>>> Sam
>>>
>>> Samuel Shelton, Ph.D.
>>> Postdoctoral Researcher,
>>> The Oldham Lab,
>>> The institute of Regeneration Medicine,
>>> University of California San Francisco,
>>> San Francisco, CA 94143
>>>
>>> email: sheltons at stemcell.ucsf.edu
>>>
>>>
>>> [[alternative HTML version deleted]]
>>>
>>> ______________________________________________
>>> R-devel at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>
>
More information about the R-devel
mailing list