[Rd] Retrieving data from aspx pages
jose ramon mazaira
jramaza at gmail.com
Wed Oct 31 17:14:54 CET 2012
I'd like to make you note that I've discovered that package RCurl
already provides a utility that allows interaction via POST requests
with servers. In fact, the FAQ for RCurl contains specifically an
example with an aspx page:
x = postForm("http://www.fas.usda.gov/psdonline/psdResult.aspx",
style = "post",
.params = list(visited="1",
lstGroup = "all",
lstCommodity="2631000",
lstAttribute="88",
lstCountry="**",
lstDate="2011",
lstColumn="Year",
lstOrder="Commodity%2FAttribute%2FCountry"))
Check this link: http://www.omegahat.org/RCurl
However, I think that it would be more useful to automate the
interaction with servers retrieving automatically the name-value pairs
required by the server (parsing the page source code) instead of
examining in each web page the appropiate fields.
2012/10/30, Paul Gilbert <pgilbert902 at gmail.com>:
> Jose
>
> As far as getting to the data, I think the best way to do this sort of
> thing would be if the site supports a SOAP or REST interface. When they
> don't (yet) then one is faced with clicking through some pages. Python
> or Java is one way to automate the process of clicking through the
> pages. I don't know how to do that in R, but would like to know if it is
> possible.
>
> But, I guess I was confused about the part you want to improve. What I
> have works fairly smoothly parsing and passing back JSON data, converted
> from a csv file, into R. The downside is that this approach requires
> more than R to be installed on the client machine. But if the object you
> get back is ASPX, then you either need to parse it directly, or convert
> it to JSON, or something else you can deal with. I suspect that will be
> fairly specific to a particular web site, but I don't really know enough
> about ASPX to be sure.
>
> Paul
>
> On 12-10-30 01:12 PM, jose ramon mazaira wrote:
>> Thanks for your interest, Paul.
>> I've checked the source code of TSjson and I've seen that what it does
>> is to call a Python script to retrieve the data. In fact, I've already
>> done this with Java using the URLConnection class and sending the
>> requested values to fill the form.
>> However, I think it would be more useful to open a connection with R
>> and to send the requested values within R, and not through an external
>> program.
>> The application I've designed, like yours, is also page-specific
>> (i.e., designed for
>> http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/AdvancedScreener.aspx),
>> but I think that our applications would be more powerful if they were
>> able to parse the name-value pairs generated from ASPX (or of any
>> other dynamically generated web page) and ask the user to select the
>> appropiate values.
>>
>> 2012/10/30, Paul Gilbert <pgilbert902 at gmail.com>:
>>> I think RHTMLForms works if you have a single form, but I have not been
>>> able to see how to use it when you need to go through a sequence of
>>> dynamically generated forms (like you can do with Python mechanize).
>>>
>>> Paul
>>>
>>> On 12-10-30 09:08 AM, Gabriel Becker wrote:
>>>> I haven't used it extensively myself, and can't speak to it's current
>>>> state but on quick inspection RHTMLForms seems worth a look for what
>>>> you
>>>> want.
>>>>
>>>> http://www.omegahat.org/RHTMLForms/
>>>>
>>>> ~G
>>>>
>>>> On Tue, Oct 30, 2012 t 5:38 AM, Paul Gilbert <pgilbert902 at gmail.com
>>>> <mailto:pgilbert902 at gmail.com>> wrote:
>>>>
>>>> I don't know of an easy way to do this in R. I've been doing
>>>> something similar with python scripts called from R. If anyone
>>>> knows
>>>> how to do this with just R, I would appreciate hearing too.
>>>>
>>>> Paul
>>>>
>>>>
>>>> On 12-10-29 04:11 PM, jose ramon mazaira wrote:
>>>>
>>>> Hi. I'm trying to write an application to retrieve financial
>>>> data
>>>> (specially bonds data) from FINRA. The web page is served
>>>> dynamically
>>>> from an asp.net <http://asp.net> application:
>>>>
>>>>
>>>> http://cxa.gtm.__idmanagedsolutions.com/finra/__BondCenter/AdvancedScreener.__aspx
>>>>
>>>> <http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/AdvancedScreener.aspx>
>>>>
>>>> I'd like to know if it's possible to fill dynamically the web
>>>> page
>>>> form from R and, after filling it (with the issuer name),
>>>> retrieve the
>>>> web page, parse the data, and covert it to appropiate R
>>>> objects.
>>>> For example, suppose I want to search data for AT&T bonds. I'd
>>>> like to
>>>> know if it's possible, within R, to fill the page served from:
>>>>
>>>>
>>>> http://cxa.gtm.__idmanagedsolutions.com/finra/__BondCenter/AdvancedScreener.__aspx
>>>>
>>>> <http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/AdvancedScreener.aspx>
>>>>
>>>> select the "corporate" option and fill with AT&T the field for
>>>> "Issuer
>>>> name", ask the page to display the results, and retrieve the
>>>> results
>>>> for each of the bonds issued by AT&T (for example:
>>>>
>>>>
>>>> http://cxa.gtm.__idmanagedsolutions.com/finra/__BondCenter/BondDetail.aspx?ID=__MDAxOTU3Qko3
>>>>
>>>> <http://cxa.gtm.idmanagedsolutions.com/finra/BondCenter/BondDetail.aspx?ID=MDAxOTU3Qko3>)
>>>>
>>>> and parsing the data from the web page.
>>>>
>>>> Thanks in advance.
>>>>
>>>> ________________________________________________
>>>> R-devel at r-project.org <mailto:R-devel at r-project.org> mailing
>>>> list
>>>> https://stat.ethz.ch/mailman/__listinfo/r-devel
>>>> <https://stat.ethz.ch/mailman/listinfo/r-devel>
>>>>
>>>>
>>>> ________________________________________________
>>>> R-devel at r-project.org <mailto:R-devel at r-project.org> mailing list
>>>> https://stat.ethz.ch/mailman/__listinfo/r-devel
>>>> <https://stat.ethz.ch/mailman/listinfo/r-devel>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Gabriel Becker
>>>> Graduate Student
>>>> Statistics Department
>>>> University of California, Davis
>>>>
>>>
>
More information about the R-devel
mailing list