[R] Accelerating binRead
Philippe de Rochambeau
phiroc at free.fr
Sun Sep 18 11:02:59 CEST 2016
Please find below code that attempts to read ints, longs and floats from a binary file (which is a simplification of my original program).
Please disregard the R inefficiencies, such as using rbind, for now.
I’ve also included Java code to generate the binary file.
The output shows that, at one point, anInt becomes undefined. Unfortunately, I couldn’t find the correct R function to determine whether inInt is undefined or not, as is.null, is.nan, and is.infinite don’t work.
Any help would be much appreciated.
Many thanks in advance.
Philippe
———————
[1] "anInt = 1"
[1] "is.null FALSE"
[1] "is.nan FALSE"
[1] "is.infinite FALSE"
[1] "aLong = 2"
[1] "aFloat = 3.44440007209778"
[1] "--------------------------"
[1] "anInt = 2"
[1] "is.null FALSE"
[1] "is.nan FALSE"
[1] "is.infinite FALSE"
[1] "aLong = 22"
[1] "aFloat = 13.4644002914429"
[1] "--------------------------"
[1] "anInt = 3"
[1] "is.null FALSE"
[1] "is.nan FALSE"
[1] "is.infinite FALSE"
[1] "aLong = 55"
[1] "aFloat = 45.4444007873535"
[1] "--------------------------"
[1] "anInt = "
[1] "is.null FALSE"
[1] "is.nan "
[1] "is.infinite "
[1] "aLong = "
[1] "aFloat = "
[1] "--------------------------"
[,1] [,2] [,3]
[1,] 1 2 3.4444
[2,] 2 22 13.4644
[3,] 3 55 45.4444
[4,] Integer,0 Integer,0 Numeric,0
>
-----------
—————————————————————
readFile <- function(inputPath) {
URL <- file(inputPath, "rb")
PLT <- matrix(nrow=0, ncol=3)
counte <- 0
max <- 4
while (counte < max) {
anInt <- readBin(con=URL, what=integer(), size=4, n=1, endian="big")
print(paste("anInt =", anInt))
#if (! (anInt == 0)) { print(paste("empty int")); break }
print(paste("is.null ", is.null(anInt)))
print(paste("is.nan ", is.nan(anInt)))
print(paste("is.infinite ", is.infinite(anInt)))
aLong <- readBin(URL, integer(), size=8, n=1, endian="big")
print(paste("aLong =", aLong))
aFloat <- readBin(URL, numeric(), size=4, n=1, endian="big")
print(paste("aFloat =", aFloat))
print("--------------------------")
PLT <- rbind(PLT, list(anInt, aLong, aFloat))
counte <- counte + 1
} # end while
close(URL)
PLT
}
fichier <- "/Users/philippe/Desktop/datatests/data0.bin"
PLT2 <- readFile(fichier)
print(PLT2)
—————————————————————
import java.io.*;
public class Main {
Main() {
writeData();
}
public static void main(String[] args) {
new Main();
}
public void writeData() {
final String path = "/Users/philippe/Desktop/datatests/data0.bin";
DataOutputStream dos;
try {
dos = new DataOutputStream(new BufferedOutputStream(new FileOutputStream(path)));
// big endian write! ("high byte first") , see https://docs.oracle.com/javase/7/docs/api/java/io/DataOutputStream.html
dos.writeInt(1);
dos.writeLong(2L);
dos.writeFloat(3.4444F);
dos.writeInt(2);
dos.writeLong(22L);
dos.writeFloat(13.4644F);
dos.writeInt(3);
dos.writeLong(55L);
dos.writeFloat(45.4444F);
dos.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException ioe) {
ioe.printStackTrace();
}
}
}
—————————————————————
> Le 17 sept. 2016 à 20:45, Philippe de Rochambeau <phiroc at free.fr> a écrit :
>
> Hi Jim,
> this is exactly the answer I was look for. Many thanks. I didn’t R had a pack function, as in PERL.
> To answer your earlier question, I am trying to update legacy code to read a binary file with unknown size, over a network, slice up it into rows each containing an integer, an integer, a long, a short, a float and a float, and stuff the rows into a matrix.
> Best regards,
> Philippe
>
>> Le 17 sept. 2016 à 20:38, jim holtman <jholtman at gmail.com <mailto:jholtman at gmail.com>> a écrit :
>>
>> Here is an example of how to do it:
>>
>> x <- 1:10 # integer values
>> xf <- seq(1.0, 2, by = 0.1) # floating point
>>
>> setwd("d:/temp")
>>
>> # create file to write to
>> output <- file('integer.bin', 'wb')
>> writeBin(x, output) # write integer
>> writeBin(xf, output) # write reals
>> close(output)
>>
>>
>> library(pack)
>> library(readr)
>>
>> # read all the data at once
>> allbin <- read_file_raw('integer.bin')
>>
>> # decode the data into a list
>> (result <- unpack("V V V V V V V V V V d d d d d d d d d d", allbin))
>>
>>
>>
>>
>> Jim Holtman
>> Data Munger Guru
>>
>> What is the problem that you are trying to solve?
>> Tell me what you want to do, not how you want to do it.
>>
>> On Sat, Sep 17, 2016 at 11:04 AM, Ismail SEZEN <sezenismail at gmail.com <mailto:sezenismail at gmail.com><mailto:sezenismail at gmail.com <mailto:sezenismail at gmail.com>>> wrote:
>> I noticed same issue but didnt care much :)
>>
>> On Sat, Sep 17, 2016, 18:01 jim holtman <jholtman at gmail.com <mailto:jholtman at gmail.com> <mailto:jholtman at gmail.com <mailto:jholtman at gmail.com>>> wrote:
>> Your example was not reproducible. Also how do you "break" out of the
>> "while" loop?
>>
>>
>> Jim Holtman
>> Data Munger Guru
>>
>> What is the problem that you are trying to solve?
>> Tell me what you want to do, not how you want to do it.
>>
>> On Sat, Sep 17, 2016 at 8:05 AM, Philippe de Rochambeau <phiroc at free.fr <mailto:phiroc at free.fr> <mailto:phiroc at free.fr <mailto:phiroc at free.fr>>>
>> wrote:
>>
>>> Hello,
>>> the following function, which stores numeric values extracted from a
>>> binary file, into an R matrix, is very slow, especially when the said file
>>> is several MB in size.
>>> Should I rewrite the function in inline C or in C/C++ using Rcpp? If the
>>> latter case is true, how do you « readBin » in Rcpp (I’m a total Rcpp
>>> newbie)?
>>> Many thanks.
>>> Best regards,
>>> phiroc
>>>
>>>
>>> -------------
>>>
>>> # inputPath is something like http://myintranet/getData <http://myintranet/getData><http://myintranet/getData <http://myintranet/getData>>?
>>> pathToFile=/usr/lib/xxx/yyy/data.bin <http://myintranet/getData <http://myintranet/getData> <http://myintranet/getData <http://myintranet/getData>>?
>>> pathToFile=/usr/lib/xxx/yyy/data.bin>
>>>
>>> PLTreader <- function(inputPath){
>>> URL <- file(inputPath, "rb")
>>> PLT <- matrix(nrow=0, ncol=6)
>>> compteurDePrints = 0
>>> compteurDeLignes <- 0
>>> maxiPrints = 5
>>> displayData <- FALSE
>>> while (TRUE) {
>>> periodIndex <- readBin(URL, integer(), size=4, n=1,
>>> endian="little") # int (4 bytes)
>>> eventId <- readBin(URL, integer(), size=4, n=1,
>>> endian="little") # int (4 bytes)
>>> dword1 <- readBin(URL, integer(), size=4, signed=FALSE,
>>> n=1, endian="little") # int
>>> dword2 <- readBin(URL, integer(), size=4, signed=FALSE,
>>> n=1, endian="little") # int
>>> if (dword1 < 0) {
>>> dword1 = dword1 + 2^32-1;
>>> }
>>> eventDate = (dword2*2^32 + dword1)/1000
>>> repNum <- readBin(URL, integer(), size=2, n=1,
>>> endian="little") # short (2 bytes)
>>> exp <- readBin(URL, numeric(), size=4, n=1,
>>> endian="little") # float (4 bytes, strangely enough, would expect 8)
>>> loss <- readBin(URL, numeric(), size=4, n=1,
>>> endian="little") # float (4 bytes)
>>> PLT <- rbind(PLT, c(periodIndex, eventId, eventDate,
>>> repNum, exp, loss))
>>> } # end while
>>> return(PLT)
>>> close(URL)
>>> }
>>>
>>> ----------------
>>> [[alternative HTML version deleted]]
>>>
>>> ______________________________________________
>>> R-help at r-project.org <mailto:R-help at r-project.org> <mailto:R-help at r-project.org <mailto:R-help at r-project.org>> mailing list -- To UNSUBSCRIBE and more, see
>>> https://stat.ethz.ch/mailman/listinfo/r-help <https://stat.ethz.ch/mailman/listinfo/r-help><https://stat.ethz.ch/mailman/listinfo/r-help <https://stat.ethz.ch/mailman/listinfo/r-help>>
>>> PLEASE do read the posting guide http://www.R-project.org/ <http://www.r-project.org/> <http://www.r-project.org/ <http://www.r-project.org/>>
>>> posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>
>> [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-help at r-project.org <mailto:R-help at r-project.org> <mailto:R-help at r-project.org <mailto:R-help at r-project.org>> mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help <https://stat.ethz.ch/mailman/listinfo/r-help><https://stat.ethz.ch/mailman/listinfo/r-help <https://stat.ethz.ch/mailman/listinfo/r-help>>
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html <http://www.r-project.org/posting-guide.html> <http://www.r-project.org/posting-guide.html <http://www.r-project.org/posting-guide.html>>
>> and provide commented, minimal, self-contained, reproducible code.
>
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org <mailto:R-help at r-project.org> mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help <https://stat.ethz.ch/mailman/listinfo/r-help>
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html <http://www.r-project.org/posting-guide.html>
> and provide commented, minimal, self-contained, reproducible code.
[[alternative HTML version deleted]]
More information about the R-help
mailing list