[R] [External] Re: read.csv fails in R console in Ubuntu terminal but works in RStudio after R 3.6.3 upgrade to R 4.0.2?

Sam H @@m@hhh1 @end|ng |rom gm@||@com
Fri Jul 17 13:54:54 CEST 2020


Hello,

Thank you very much to you all to look into this.

I came across this problem when I was using TTR::stockSymbols() (
https://github.com/joshuaulrich/TTR/blob/e6609b9f7621f3a4b1a204c159af61aebc89997e/R/WebData.R)
.

As a workaround I added this function to my private R package and instead
of read.csv I am now using data.table::fread() which properly (without
failing) downloads the file and reads it.

Best,
Sam


On Fri, Jul 17, 2020 at 4:30 AM ruipbarradas <ruipbarradas using sapo.pt> wrote:

> Hello,
>
> Yes, I thought it's a site policy issue too. But the file can be accessed
> and read/downloaded from RStudio and Firefox so apparently there's no
> reason why R console shouldn't .
>
> Anyway, I believe it's time for the OP to say someyhing, maybe he has
> solved it and there's no point in continuing.
>
> Rui Barradas
>
>
>
>
>
> Enviado a partir do meu smartphone Samsung Galaxy.
> -------- Mensagem original --------
> De: luke-tierney using uiowa.edu
> Data: 17/07/2020 02:59 (GMT+00:00)
> Para: Ista Zahn <istazahn using gmail.com>
> Cc: Rui Barradas <ruipbarradas using sapo.pt>, r-help using r-project.org, Sam H <
> sam.hhh1 using gmail.com>
> Assunto: Re: [External] Re: [R] read.csv fails in R console in Ubuntu
> terminal but works in RStudio after R 3.6.3 upgrade to R 4.0.2?
>
> On my Ubuntu system the download with read.csv succeeds in an R
> console if I set the HTTPUserAgent and download.file.method options to
> match the ones used by RStudio.
>
> Given how picky the server is being I would worry about whether this
> use is in line with the site's terms of service.
>
> Best,
>
> luke
>
> On Thu, 16 Jul 2020, Ista Zahn wrote:
>
> > On Thu, Jul 16, 2020 at 5:15 PM Ista Zahn <istazahn using gmail.com> wrote:
> >>
> >> On Thu, Jul 16, 2020 at 8:18 AM Rui Barradas <ruipbarradas using sapo.pt>
> wrote:
> >>>
> >>> Hello,
> >>>
> >>> Thanks, but no, download.file still gives 403 Forbidden with both
> method
> >>> = "libcurl" and method = "wget".
> >>
> >> I think that makes it "not an R question". Ask on
> >> https://unix.stackexchange.com/ maybe?
> >
> > Oh, sorry I misread your message. Nevertheless:
> >
> > $ curl "
> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download
> "
> > <HTML><HEAD>
> > <TITLE>Access Denied</TITLE>
> > </HEAD><BODY>
> > <H1>Access Denied</H1>
> >
> > You don't have permission to access
> >
> "http://old.nasdaq.com/screening/companies-by-name.aspx?"
> > on this server.<P>
> > Reference #18.5506d217.1594934303.938edcb
> > </BODY>
> > </HTML>
> >
> > $ wget "
> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download
> "
> > --2020-07-16 17:19:12--
> >
> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download
> > Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
> > Resolving old.nasdaq.com (old.nasdaq.com)... 2600:1400:9000:28f::1b46,
> > 2600:1400:9000:29b::1b46, 23.78.161.120
> > Connecting to old.nasdaq.com
> > (old.nasdaq.com)|2600:1400:9000:28f::1b46|:443... connected.
> > HTTP request sent, awaiting response... 403 Forbidden
> > 2020-07-16 17:19:12 ERROR 403: Forbidden.
> >
> > I don't think this is an R problem.
> >
> > Best,
> > Ista
> >
> >>
> >> Best,
> >> Ista
> >>
> >>>
> >>> Rui Barradas
> >>>
> >>> Às 05:31 de 16/07/20, Jeff Newmiller escreveu:
> >>>> Perhaps read FAQ 7.43? [1]
> >>>>
> >>>> [1]
> https://cran.r-project.org/doc/FAQ/R-FAQ.html#How-can-I-enable-secure-https-downloads-in-R_003f
> >>>>
> >>>> On July 15, 2020 4:02:27 PM PDT, Rui Barradas <ruipbarradas using sapo.pt>
> wrote:
> >>>>> Hello,
> >>>>>
> >>>>> R 4.0.2 on Ubuntu 20.04 LTS, sessionInfo below.
> >>>>>
> >>>>> I'm also unable to read the file with Rscript from the Ubuntu
> terminal
> >>>>> but the error is not the same as the OP's.
> >>>>>
> >>>>>
> >>>>> The first try was a file test1.R with the following commands.
> >>>>>
> >>>>> x<-"
> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download
> "
> >>>>> read.csv(x, as.is=TRUE, na="n/a")
> >>>>>
> >>>>>
> >>>>> And run with Rscript
> >>>>>
> >>>>> rui using rui:~$ Rscript --vanilla test1.R
> >>>>> Error in file(file, "rt") :
> >>>>>    cannot open the connection to
> >>>>> '
> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download
> '
> >>>>> Calls: read.csv -> read.table -> file
> >>>>> In addition: Warning message:
> >>>>> In file(file, "rt") :
> >>>>>    cannot open URL
> >>>>> '
> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download
> ':
> >>>>>
> >>>>> HTTP status was '403 Forbidden'
> >>>>> Execution halted
> >>>>>
> >>>>>
> >>>>>
> >>>>> The second try was download.file() and then read it.
> >>>>> File test2.R is:
> >>>>>
> >>>>> x<-"
> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download
> "
> >>>>> download.file(x, "companylist.csv")
> >>>>> read.csv("companylist.csv", as.is=TRUE, na="n/a")
> >>>>>
> >>>>>
> >>>>> But this too failed with error 403 Forbiden.
> >>>>>
> >>>>> rui using rui:~$ Rscript --vanilla test2.R
> >>>>> trying URL
> >>>>> '
> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download
> '
> >>>>> Error in download.file(x, "companylist.csv") :
> >>>>>    cannot open URL
> >>>>> '
> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download
> '
> >>>>> In addition: Warning message:
> >>>>> In download.file(x, "companylist.csv") :
> >>>>>    cannot open URL
> >>>>> '
> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download
> ':
> >>>>>
> >>>>> HTTP status was '403 Forbidden'
> >>>>> Execution halted
> >>>>>
> >>>>>
> >>>>> This is my session info.
> >>>>>
> >>>>> rui using rui:~$ Rscript --vanilla -e 'sessionInfo()'
> >>>>> R version 4.0.2 (2020-06-22)
> >>>>> Platform: x86_64-pc-linux-gnu (64-bit)
> >>>>> Running under: Ubuntu 20.04 LTS
> >>>>>
> >>>>> Matrix products: default
> >>>>> BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
> >>>>> LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0
> >>>>>
> >>>>> locale:
> >>>>>   [1] LC_CTYPE=pt_PT.UTF-8       LC_NUMERIC=C
> >>>>>   [3] LC_TIME=pt_PT.UTF-8        LC_COLLATE=pt_PT.UTF-8
> >>>>>   [5] LC_MONETARY=pt_PT.UTF-8    LC_MESSAGES=pt_PT.UTF-8
> >>>>>   [7] LC_PAPER=pt_PT.UTF-8       LC_NAME=C
> >>>>>   [9] LC_ADDRESS=C               LC_TELEPHONE=C
> >>>>> [11] LC_MEASUREMENT=pt_PT.UTF-8 LC_IDENTIFICATION=C
> >>>>>
> >>>>> attached base packages:
> >>>>> [1] stats     graphics  grDevices utils     datasets  methods   base
> >>>>>
> >>>>> loaded via a namespace (and not attached):
> >>>>> [1] compiler_4.0.2
> >>>>>
> >>>>>
> >>>>>
> >>>>> Às 08:45 de 15/07/20, Sam H escreveu:
> >>>>>> Hi,
> >>>>>>
> >>>>>> I am trying to download some data using read.csv and it works
> >>>>> perfectly in
> >>>>>> RStudio and fails in the R console in the terminal in Ubuntu 18.04
> >>>>> after
> >>>>>> upgrading from R 3.6.3 to 4.0.2. Before upgrading this worked in the
> >>>>> R
> >>>>>> console in the terminal also without any issues.
> >>>>>>
> >>>>>> Why would that be? How to fix this?
> >>>>>>
> >>>>>> Below please find R code output and sessionInfo().
> >>>>>>
> >>>>>> *Works in RStudio*
> >>>>>>
> >>>>>>>
> >>>>> read.csv("
> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download
> ",
> >>>>> header=TRUE, as.is=TRUE, na="n/a")
> >>>>>>       Symbol                                                Name
> >>>>>> LastSale MarketCap IPOyear1      TXG
> >>>>>> 10x Genomics, Inc.  87.4400     $8.6B    20192       YI
> >>>>>>                              111, Inc.   6.4800  $533.69M    20183
> >>>>>> PIH              1347 Property Insurance Holdings, Inc.   4.5350
> >>>>>> $27.52M    2014
> >>>>>>    sessionInfo()
> >>>>>> R version 4.0.2 (2020-06-22)
> >>>>>> Platform: x86_64-pc-linux-gnu (64-bit)
> >>>>>> Running under: Ubuntu 18.04.4 LTS
> >>>>>>
> >>>>>> Matrix products: default
> >>>>>> BLAS:   /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
> >>>>>> LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so
> >>>>>>
> >>>>>> locale:
> >>>>>>    [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
> >>>>>> LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
> >>>>>>    [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
> >>>>>> LC_PAPER=en_US.UTF-8       LC_NAME=C
> >>>>>>    [9] LC_ADDRESS=C               LC_TELEPHONE=C
> >>>>>> LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
> >>>>>>
> >>>>>> attached base packages:[1] stats     graphics  grDevices utils
> >>>>>> datasets  methods   base
> >>>>>>
> >>>>>> loaded via a namespace (and not attached):[1] compiler_4.0.2
> >>>>> tools_4.0.2
> >>>>>>
> >>>>>> *Fails in R console in terminal*
> >>>>>>
> >>>>>>      >
> >>>>> read.csv("
> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download
> ",
> >>>>>> header=TRUE, as.is=TRUE, na="n/a")
> >>>>>> Error in file(file, "rt") :
> >>>>>>     cannot open the connection to
> >>>>>>
> >>>>> '
> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download
> '
> >>>>>> In addition: Warning message:
> >>>>>> In file(file, "rt") :
> >>>>>>     URL
> >>>>> '
> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download
> ':
> >>>>>> status was 'Failure when receiving data from the peer'>
> traceback()3:
> >>>>>> file(file, "rt")2: read.table(file = file, header = header, sep =
> >>>>> sep,
> >>>>>> quote = quote,
> >>>>>>          dec = dec, fill = fill, comment.char = comment.char, ...)1:
> >>>>>>
> >>>>> read.csv("
> https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download
> ",
> >>>>>>          header = TRUE, as.is = TRUE, na = "n/a")>  sessionInfo()
> >>>>>> R version 4.0.2 (2020-06-22)
> >>>>>> Platform: x86_64-pc-linux-gnu (64-bit)
> >>>>>> Running under: Ubuntu 18.04.4 LTS
> >>>>>>
> >>>>>> Matrix products: default
> >>>>>> BLAS:   /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
> >>>>>> LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so
> >>>>>>
> >>>>>> locale:
> >>>>>>    [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
> >>>>>>    [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
> >>>>>>    [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
> >>>>>>    [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
> >>>>>>    [9] LC_ADDRESS=C               LC_TELEPHONE=C            [11]
> >>>>>> LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
> >>>>>>
> >>>>>> attached base packages:[1] stats     graphics  grDevices utils
> >>>>>> datasets  methods   base
> >>>>>>
> >>>>>> loaded via a namespace (and not attached):[1] compiler_4.0.2>
> >>>>>>
> >>>>>> I also asked this question here
> >>>>>>
> >>>>>
> https://stackoverflow.com/questions/62898008/why-read-csv-fails-in-r-console-in-ubuntu-terminal-but-works-in-rstudio-after-r
> >>>>>> . Since there was no answer on stackoverflow I sent this question
> >>>>> also to
> >>>>>> this list.
> >>>>>>
> >>>>>> Best regards,
> >>>>>> Sam
> >>>>>>
> >>>>>>     [[alternative HTML version deleted]]
> >>>>>>
> >>>>>> ______________________________________________
> >>>>>> R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> >>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>>>>> PLEASE do read the posting guide
> >>>>> http://www.R-project.org/posting-guide.html
> >>>>>> and provide commented, minimal, self-contained, reproducible code.
> >>>>>>
> >>>>>
> >>>>> ______________________________________________
> >>>>> R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> >>>>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>>>> PLEASE do read the posting guide
> >>>>> http://www.R-project.org/posting-guide.html
> >>>>> and provide commented, minimal, self-contained, reproducible code.
> >>>>
> >>>
> >>> ______________________________________________
> >>> R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> >>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> >>> and provide commented, minimal, self-contained, reproducible code.
> >
> > ______________________________________________
> > R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
> >
>
> --
> Luke Tierney
> Ralph E. Wareham Professor of Mathematical Sciences
> University of Iowa                  Phone:             319-335-3386
> Department of Statistics and        Fax:               319-335-3017
>     Actuarial Science
> 241 Schaeffer Hall                  email:   luke-tierney using uiowa.edu
> Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu
>


-- 
Best regards,
Sam

	[[alternative HTML version deleted]]



More information about the R-help mailing list