[R] RE : R getting slower until it breaks...

Bastien Ferland-Raymond bastien.ferland-raymond.1 at ulaval.ca
Wed Oct 6 21:06:35 CEST 2010


Thanks a lot for your quick answer.  Here is my answer to your questions:

Have you looked to see how fast your memory might be growing?  
BFR- Yes I did, it's not to bad, it's starts around 60 000ko, rise up to 120 000 at the most, so not too scary.

Are you leaving around any large objects that should be removed? 
BFR- I was carefull making sure the function doesn't create anything that would be visible with objects().  Could it be creating other type (hidden) objects?  Maybe, but I'm not very familliar with that stuff.

Have you looked to see if you are paging? 
BFR- I just red the wiki about paging, didn't know that term before.  If I look at perfmon, its looks like keeping steady at 6000 pages/s with rare peaks as high as 900 000. Does it sounds normal?  How can it affect R?

Is it your CPU time that is increasing, or your wall clock time?
BFR- If I go to the task manager - performance.  R is initially using around 40% of the processor (so around 80% of 1 core) but with (real) time passing, it gets lower and lower to get as low as 6% (12% of one core).  I was surprized to see that as usually my simulation in R use one whole core.  

It sounds like there might be some memory leak that might be causing your process size to grow and possibly causing paging.  You will need to gather some of the performance data that perfmon can provide and look at the memory usage, CPU time and I/O rates over time to see if there are any changes.
BFR- The term "Memory leak" feels right with my problem.  Is there ways I can control/detect/prevent this kind of problem in R.  Also, how can I check the I/O, i never looked at that before.

Thanks again

Bastien



On Wed, Oct 6, 2010 at 2:11 PM, Bastien Ferland-Raymond
<bastien.ferland-raymond.1 at ulaval.ca> wrote:
> Hello R-users,
>
> I'm currently facing a pretty hard problem which I'm hopping you'll be able to help me with.  I'm using R to create images.  That alone is not the problem, the problem is that I'm using R to create 168 000 images...  My code (which is given below) use different package (raster and rgdal) to import a image (size 20gig) and divide it into 168 000 pictures that are 100 pixel x 100 pixel.  The code works fine for making the images, but if I ask it to run all 168 000, it always breaks around 15 000.
>
> It starts with the code being able to make around 2 pictures per second, but then it slows down and after around 2000 pictures it's only 1 picture per second.  Later on it's getting closer to 1 pictures every 3 seconds etc.  until it bugs.  I have no error message, only Windows that tells me that "R encounter a problem and most be close..."  Initially I though it was a Windows problem, that I couldn't put too many file into a folder and it was slowering it down.  Then I divided my batch process into smaller (5000 files) folder but it didn't help, still breaks at 15 000.  I also try to do gc() after each 5000 pictures to save memory but it didn't help either.  I removed every loops from the code because I thought it was the problem, but it was just faster at bugging... After the bug, I need to restart the computer if I want to go back to the initial speed.
>
> I'm pretty much running out of options.  It's there limitation in R as the number of files it can create in one session?  Is it a windows problem?  Is there better way to clear the memory than gc()? Any thought on that?
>
> I'm using R 2.11.1, win XP, my hard drive is NTSF, computer: intel core2 duo E6750 32 bit with 2 gig of Ram.
>
> Here is my code, but I doubt it would help much with my problem:
>
> ########
> # It made of 4 functions (sorry, it's french):
>
> ##########################################################################
> ##########################################################################
> ###  Ensemble des fonctions pour faire les images NDVI rouge et verte  ###
> ##########################################################################
> ######  Bastien Ferland-Raymond, 5 oct 2010  #############################
> ##########################################################################
>
> ########
> ## Simplement rouler le script au complet
> ########
> ### Library nécessaire:
> library(raster)
> library(rgdal)
> library(shapefiles)
>
> #############################################################################
> ## Fonction 1  -  NDVI a partir de coordonnee Pixel et largeur #####
>  calculate_NDVI<- function(Type, object, VALUE) {
>   redorgreen <- ifelse(Type=="red",2,3)
>   list1 <- unstack(object)
>   rast1 <- list1[[1]]
>   rast2 <- list1[[redorgreen]]
>   NAvalue(rast1)<- -99999
>   NAvalue(rast2)<- -99999
>   cells1 <- getValuesBlock(rast1,row=VALUE[[2]],nrow=VALUE[[3]],col=VALUE[[1]],ncol=VALUE[[3]])
>   cells2 <- getValuesBlock(rast2,row=VALUE[[2]],nrow=VALUE[[3]],col=VALUE[[1]],ncol=VALUE[[3]])
>   cells1[is.na(cells1)]<-0;
>   cells2[is.na(cells2)]<-0;
>   calculNDVI <-(cells1 - cells2) / (cells1 + cells2)
>   NDVImatrix <- matrix(calculNDVI,nrow=VALUE[[3]],ncol=VALUE[[3]], byrow=TRUE)
>   NDVImatrix <- NDVImatrix + 1
>   NDVImatrix <- NDVImatrix * (255/2)
>   return(NDVImatrix)
>   }
> #################################################################################
> ## Fonction 1b  -  Faire le tiff
>  make.tiff<- function(NV=newValues,TT=Type,img=imgRaster,nom){
>  pixelNDVIMatrix <- calculate_NDVI(TT,img,NV[c(1,2,3)])
>  newRaster <- raster(pixelNDVIMatrix)
>  NAvalue(newRaster)<-999999
>  nnom<-nom[NV[4]]
>  writeRaster(newRaster, filename=nnom,datatype="INT1U",format="GTiff",overwrite=FALSE)
>  aaa<-2
> }
>
> #################################################################################
> ## Fonction 2  -  Creation de fonction convertissant les coordonnee metrique en coordonnee pixels #####
>  latlong_to_pixels<- function(Coord, facteur, meterWidth=NULL) {   #Coord doit être c(x,y)
>  newX <- Coord[1] / facteur
>  newY <- Coord[2] / facteur
>  if(!is.null(meterWidth)){
>   newWidth <- meterWidth / facteur
>   return(c(newX,newY,newWidth))
>  }
>  return(c(newX,newY))
>  }
>
> #############################################################################
> ####  Fonction 3  -  Fonction principal   #####
>  make.NDVI.photo<- function(tableDesPlacettes,Type,newImagesDirectory,textAndImgDirectory="U:\\kNN_Valcartier\\Photo aerienne"){
>  lastWD<- getwd()
> setwd(textAndImgDirectory)
>  imgRaster<- stack(imageAssociee)
>  x1<- tableDesPlacettes[,2] - xmin(imgRaster) - (tailleFenetres/2)     # The image origin for calculation is in the top left corner
>  y1<- ymax(imgRaster) - tableDesPlacettes[,3] - (tailleFenetres/2)
> coo <- cbind(x1,y1)
>  newValues<- t(apply(coo,1,latlong_to_pixels,facteurMetreParPixel,tailleFenetres))
>  newImgName<- paste(newImagesDirectory,substr(Type,1,3),"_","GC",tableDesPlacettes[,1],".tif",sep="")
> apply(cbind(newValues,1:length(newImgName)),1,make.tiff,Type,imgRaster,nom=newImgName)
> setwd(lastWD)
> }
>
>
> ###########################
> ## Executing fonctions:
> #############################
>
>
> ###  loader les données brutes de fenêtres
> fichier.fenetre.brute<-read.dbf("U:\\kNN_Valcartier\\Fenêtres 30x30 1 octobre\\168700_30m_centroid.dbf", header=T)
> ###  Sélectionner les fenêtres complètes
> fenetre.complete<-round(fichier.fenetre.brute[[1]][,2],1)==900
> ###  Sortir les centroides pour extraction
> centro.tout.900<-fichier.fenetre.brute[[1]][fenetre.complete,c(1,5,6)]
> #rm(fichier.fenetre.brute) ; gc()
>
> ## données nécessaires pour la fonction
>  imageAssociee<- "mosaique_all_v1.img"  # nom de l'image
>  facteurMetreParPixel<- 0.3         # combien de metre vaut un pixel
>  tailleFenetres= 30            # en metre
>
>  start.time<-Sys.time();start.time
> make.NDVI.photo(centro.tout.900[19137:24136,],"red","BFR\\NDVI_red_fenetre\\batch 3\\")
>  stop.time<-Sys.time()
>  time.run<-stop.time-start.time
>  alarm()
>  time.run
> gc()
>  start.time<-Sys.time();start.time
> make.NDVI.photo(centro.tout.900[24137:29136,],"red","BFR\\NDVI_red_fenetre\\batch 4\\")
>  stop.time<-Sys.time()
>  time.run<-stop.time-start.time
>  alarm()
>  time.run
> gc()
>
> #############
>
>
> Voilà,
>
> Thanks!
>
> Bastien Ferland-Raymond
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>



--
Jim Holtman
Cincinnati, OH
+1 513 646 9390

What is the problem that you are trying to solve?


More information about the R-help mailing list