[R] hdf5 package segfault when processing large data

[Marcus Daniels]

Hi Budi and Bill,

As a workaround, you could try setting your stack limit to a larger 
limit (ulimit -s).   I'll change that to a heap allocation.

Thanks,

Marcus

William Dunlap wrote:
> This is probably due to the code in hdf5.c allocating a huge
> matrix, buf, on the stack with
>
>     883           unsigned char buf[rowcount][size];
>
> It dies with the segmentatio fault (stack overflow, in particular)
> at line 898, where it tries to access this buf.
>
>     885           for (ri = 0; ri < rowcount; ri++)
>     886             for (pos = 0; pos < colcount; pos++)
>     887               {
>     888                 SEXP item = VECTOR_ELT (val, pos);
>     889                 SEXPTYPE type = TYPEOF (item);
>     890                 void *ptr = &buf[ri][offsets[pos]];
>     891
>     892                 switch (type)
>     893                   {
>     894                   case REALSXP:
>     895                     memcpy (ptr, &REAL (item)[ri], sizeof
> (double));
>     896                     break;
>     897                   case INTSXP:
>     898                     memcpy (ptr, &INTEGER (item)[ri], sizeof
> (int));
>     899                     break;
>
> The code should use one of the allocators in the R API instead
> of putting the big memory block on the stack.
>
>   
>> ========
>> The data example, the list continue for more than 250,000 
>> rows: sample.txt
>> ========
>> Date	Time	f1	f2	f3	f4	f5
>> 20070328	07:56	463	463.07	462.9	463.01	1100
>> 20070328	07:57	463.01	463.01	463.01	463.01	200
>> ....
>>
>> ______________________________________________
>>