[BioC] EBImage Error in .Call("lib_readImages", as.character(files), as.integer(colormode),
Hervé Pagès
hpages at fhcrc.org
Sat Apr 9 02:42:29 CEST 2011
Hi Greg,
Seems like you have an integer overflow at the C level (src/io.c,
line 276):
int width, height, nplanes;
image = PROTECT(allocVector(REALSXP, width * height * nplanes));
For big values of width and/or height and/or nplanes, the result of
the product will be a negative integer.
The way they catch integer multiplication overflow in the R base code
is with something like (found in src/main/array.c):
if (nrow < 0 || ncol < 0)
error(_("negative extents to matrix"));
if ((double)nrow * (double)ncol > INT_MAX)
error(_("allocMatrix: too many elements specified"));
n = nrow * ncol;
PROTECT(s = allocVector(mode, n));
The code above is used in the allocMatrix() function. They also use
the same trick in alloc3DArray():
if (nrow < 0 || ncol < 0 || nface < 0)
error(_("negative extents to 3D array"));
if ((double)nrow * (double)ncol * (double)nface > INT_MAX)
error(_("alloc3Darray: too many elements specified"));
n = nrow * ncol * nface;
PROTECT(s = allocVector(mode, n));
Although this works fine in practice, this is not in theory 100%
portable (it relies on some subtle assumptions about the size of
the "int" type in C and how INT_MAX would accurately compare to a
a double).
Another common technique is to use a type of integer that has at
least twice the size of the "int" type. Note that, according to
the ANSI C specs, the "long int" type is not guaranteed to satisfy
this. Not even the "long long int". But it seems that in practice,
and probably on most (if not all) platforms where R is supported,
"int" is 32-bit and "long long int" is 64-bit. So the following
should be reasonably portable:
int width, height, nplanes;
unsigned long long n;
n = width;
n *= height;
if (n > INT_MAX)
error("readCellomics: image is too big");
n *= nplanes;
if (n > INT_MAX)
error("readCellomics: image is too big")
image = PROTECT(allocVector(REALSXP, n));
The advantage of this solution over the "double" solution above is
that the multiplication is performed only once.
Note that INT_MAX is defined in limits.h
A 3rd solution which seems to be cleaner is to just use unsigned
int. Since "modulo arithmetic" is well defined on unsigned int
(according to the C99 standard), one can do:
int width, height, nplanes, n;
unsigned int uwidth, uheight, unplanes, n1, n2;
uwidth = width;
uheight = height;
n1 = uwidth * uheight;
if ((uheight != 0U) && (n1 / uheight != uwidth))
error("readCellomics: image is too big");
unplanes = nplanes;
n2 = n1 * unplanes;
if ((unplanes != 0U) && (n2 / unplanes != n1))
error("readCellomics: image is too big");
n = n2;
image = PROTECT(allocVector(REALSXP, n));
But why not just use alloc3DArray() or allocMatrix() in your code in
the first place. They will not only catch the integer overflow problem
for you, they will also set the dim attributes on 'image'.
Hope this helps,
H.
On 11-04-08 12:17 AM, Gregoire Pau wrote:
> This is right, Henrik.
>
> The image is 52703x20688: this is huge and I cannot display it on my
> computer. This image would take 52703x20688x8 bytes in memory, i.e.
> about 8 GBytes. I am not sure if R (even the 64-bit version of R) can
> handle such big objects.
>
> Greg
> ---
> Gregoire Pau
> EMBL Research Officer
> http://www.embl.de/~gpau/
>
> On 07/04/11 18:50, Vladimir Morozov wrote:
>> Henrik,
>>
>> I think you make the right guess.
>>
>> Anyway the tiff file is
>> 0500101820.0.tif -
>> https://docs.google.com/leaf?id=0B2NLCeM3RRN2ODIxYWVmZjItYzYwNS00YjJjLWE2YjQtYjQwZGUzZTJkOTBi&hl=en
>>
>>
>> Thanks,
>>
>> Vlad
>>
>>
>> -----Original Message-----
>> From: henrik.bengtsson at gmail.com [mailto:henrik.bengtsson at gmail.com]
>> On Behalf Of Henrik Bengtsson
>> Sent: Thursday, April 07, 2011 12:46 PM
>> To: Vladimir Morozov
>> Cc: Gregoire Pau; bioconductor at stat.math.ethz.ch
>> Subject: Re: [BioC] EBImage Error in .Call("lib_readImages",
>> as.character(files), as.integer(colormode),
>>
>> A guess: Your image dimension might be too large. In R,
>> matrices/arrays are stored as vectors with a dimension attribute
>> meaning the number of elements is limited by prod(dim(x)) for an array
>> 'x', not max(dim(x)). Since indices are 32-bit integers in R, you may
>> be hitting the maximum limit:
>>
>>> .Machine$integer.max
>> [1] 2147483647
>>
>> # For your small image that works
>>> dim<- c(26352,10344,3)
>>> prod(dim)
>> [1] 817755264
>>> (prod(dim)<= .Machine$integer.max)
>> [1] TRUE
>>
>> # For the 90/50=1.8 times larger one
>>> dim<- 1.8*c(26352,10344,3)
>>> prod(dim)
>> [1] 4769148700
>>> (prod(dim)<= .Machine$integer.max)
>> [1] FALSE
>>
>> /Henrik
>>
>>
>> On Thu, Apr 7, 2011 at 9:23 AM, Vladimir Morozov<vmorozov at als.net> wrote:
>>> Hi Gregory,
>>>
>>> It dosen't seem to be multiple version problem:
>>>
>>> [rstats:ImageAnal] ldd
>>> /usr/local/lib64/R/library/EBImage/libs/EBImage.so|grep -i magic
>>> libMagickCore.so.3 => /usr/lib64/libMagickCore.so.3
>>> (0x00007f264092f000)
>>> libMagickWand.so.3 => /usr/lib64/libMagickWand.so.3
>>> (0x00007f263e62c000) [rstats:ImageAnal] which convert /usr/bin/convert
>>> [rstats:ImageAnal] ldd /usr/bin/convert
>>> linux-vdso.so.1 => (0x00007fff09bff000)
>>> libMagickCore.so.3 => /usr/lib64/libMagickCore.so.3
>>> (0x00007fdad3f45000)
>>> libMagickWand.so.3 => /usr/lib64/libMagickWand.so.3
>>> (0x00007fdad3c26000)
>>> ....
>>>
>>>
>>> I can resize the tiff image with ImageMagic:
>>> convert -channel B -resize '50%'
>>> /data/projects/ImageAnal/tiff/0500101820.0.tif
>>> /data/projects/ImageAnal/tiff/0500101820.05.png
>>> And then get it into R :
>>>> x<- readImage('/data/projects/ImageAnal/tiff/0500101820.05.png')
>>>> dim(x)
>>> [1] 26352 10344 3
>>>
>>> However bigger resolution images fail:
>>> convert -channel B -resize '90%'
>>> /data/projects/ImageAnal/tiff/0500101820.0.tif
>>> /data/projects/ImageAnal/tiff/0500101820.09.png
>>>
>>>> x<- readImage('/data/projects/ImageAnal/tiff/0500101820.09.png')
>>> Error in .Call("lib_readImages", as.character(files),
>>> as.integer(colormode), :
>>> negative length vectors are not allowed
>>>
>>> So it doesn't look as ImageMagic problem. If you still want the tiff
>>> file, I can post it
>>>
>>> Best
>>> Vlad
>>>
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Gregoire Pau [mailto:gregoire.pau at embl.de]
>>> Sent: Thursday, April 07, 2011 11:53 AM
>>> To: Vladimir Morozov
>>> Cc: 'bioconductor at stat.math.ethz.ch'
>>> Subject: Re: [BioC] EBImage Error in .Call("lib_readImages",
>>> as.character(files), as.integer(colormode),
>>>
>>> Hello Vladimir,
>>>
>>> If ImageMagick is able to read the image, EBImage should be able to
>>> do so. Do you have several installed versions of ImageMagick ? To
>>> which ImageMagick version EBImage is linked to (try otool or ldd on
>>> EBImage.so or EBImage.dll) ?
>>>
>>> TIFF is a not a file format but a file container which can contain
>>> exotic formats: could you please send me your image for inspection ?
>>>
>>> Cheers,
>>>
>>> Greg
>>> ---
>>> Gregoire Pau
>>> EMBL Research Officer
>>> http://www.embl.de/~gpau/
>>>
>>> On 07/04/11 16:48, Vladimir Morozov wrote:
>>>> Hi,
>>>>
>>>> I get image reading error:
>>>>
>>>>> x<- readImage(
>>>>
>>>> "/data/projects/ImageAnal/tiff/0500101820.0.tif"
>>>>
>>>> )
>>>>
>>>> Error in .Call("lib_readImages", as.character(files),
>>>> as.integer(colormode), :
>>>>
>>>> negative length vectors are not allowed
>>>>
>>>>> packageVersion('EBImage')
>>>>
>>>> [1] '3.6.0'
>>>>
>>>>>
>>>>
>>>> ImageMagic can work with this image:
>>>>
>>>> convert -channel B -resize '70%'
>>>> /data/projects/ImageAnal/tiff/0500101820.0.tif
>>>> /data/projects/ImageAnal/tiff/0500101820.07.jpeg&
>>>>
>>>>
>>>>
>>>> Tthis image is the bigest image from SVS stack:
>>>>
>>>> convert /data/projects/ImageAnal/svs/0500101820.svs
>>>> /data/projects/ImageAnal/tiff/0500101820.%d.tif
>>>>
>>>> [rstats:ImageAnal] ll -h /data/projects/ImageAnal/tiff/0500101820.*
>>>> -rw-r--r-- 1 vmorozov users 24K 2011-04-06 15:18
>>>> /data/projects/ImageAnal/tiff/0500101820.1.tif
>>>> -rw-r--r-- 1 vmorozov users 81M 2011-04-06 15:18
>>>> /data/projects/ImageAnal/tiff/0500101820.0.tif
>>>> -rw-r--r-- 1 vmorozov users 155K 2011-04-06 15:18
>>>> /data/projects/ImageAnal/tiff/0500101820.5.tif
>>>> -rw-r--r-- 1 vmorozov users 49K 2011-04-06 15:18
>>>> /data/projects/ImageAnal/tiff/0500101820.4.tif
>>>> -rw-r--r-- 1 vmorozov users 196K 2011-04-06 15:18
>>>> /data/projects/ImageAnal/tiff/0500101820.3.tif
>>>> -rw-r--r-- 1 vmorozov users 3.0M 2011-04-06 15:18
>>>> /data/projects/ImageAnal/tiff/0500101820.2.tif
>>>> [rstats:ImageAnal] convert -v
>>>> Version: ImageMagick 6.6.1-0 2010-07-29 Q16
>>>> http://www.imagemagick.org
>>>> Copyright: Copyright (C) 1999-2010 ImageMagick Studio LLC
>>>> Features: OpenMP
>>>> .....
>>>>
>>>>
>>>>
>>>> I have the same problem with other SVS stacks, i.e readImage fails to
>>>> read the most quality images. it works fine on smaller images
>>>>
>>>>
>>>>
>>>> Any suggestions would be apriciated
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>> Vladimir Morozov
>>>> Sr. Computational Biologist
>>>> ALS Therapy Development Institute
>>>> 215 First Street, Cambridge MA, 02142 www.als.net<http://www.als.net>
>>>> Help End ALS! Click Here<http://als.net/EndALS.aspx> or Text
>>>> "EndALS" to 20222 to make a $5 donation towards research.
>>>>
>>>>
>>>>
>>>> *********************************************************************
>>>> *
>>>> ***************** The information contained in this electronic
>>>> message is ...{{dropped:21}}
>>>>
>>>> _______________________________________________
>>>> Bioconductor mailing list
>>>> Bioconductor at r-project.org
>>>> https://stat.ethz.ch/mailman/listinfo/bioconductor
>>>> Search the archives:
>>>> http://news.gmane.org/gmane.science.biology.informatics.conductor
>>>
>>>
>>> **********************************************************************
>>> ***************** The information contained in this electronic message
>>> is ...{{dropped:18}}
>>>
>>> _______________________________________________
>>> Bioconductor mailing list
>>> Bioconductor at r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/bioconductor
>>> Search the archives:
>>> http://news.gmane.org/gmane.science.biology.informatics.conductor
>>>
>>
>>
>> ***************************************************************************************
>>
>> The information contained in this electronic message i...{{dropped:3}}
>
> _______________________________________________
> Bioconductor mailing list
> Bioconductor at r-project.org
> https://stat.ethz.ch/mailman/listinfo/bioconductor
> Search the archives:
> http://news.gmane.org/gmane.science.biology.informatics.conductor
--
Hervé Pagès
Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M2-B876
P.O. Box 19024
Seattle, WA 98109-1024
E-mail: hpages at fhcrc.org
Phone: (206) 667-5791
Fax: (206) 667-1319
More information about the Bioconductor
mailing list