topic Extracting data from webpage in QlikView

Extracting data from webpage

Thu, 14 Apr 2011 09:38:08 GMT

Hi,

I am trying to extract the data from the webpage i am getting the error as not aurthorized to access.I understood the actual problem is Internet is accessed based on credentials and the data in the webpage also accessed through passing the credentials.

(a)Websites are accessed by providing the username and Password.

(b)Data in the website are accessed by logining in to the specific website.

How the same can be implemented. Please suggest.

Extracting data from webpage

stephencredmond — Thu, 14 Apr 2011 09:47:11 GMT

QlikView, I believe, uses IE libraries to connect over http. This means that it passes Windows authentication to "intranet" sites (e.g. Sharepoint) correctly. However, if there is some sort of challenge/response or forms authentication, then that can't be handled.

The alternative is to use an external tool to connect to the website and download the data locally and then consume it in QlikView. For example, I have previously written a simple c# application to connect to a sharepoint site and download the contents locally to load into QlikView. There are many freeware applications available that can do this type of thing.

Regards,

Stephen

Extracting data from webpage

gauravkhare — Thu, 14 Apr 2011 10:01:28 GMT

Hi XXX,

you can find the solution in one of the sample applications given in QlikView 9.0 under currency conversion, in which they are extracting data from a websites containing the exchange rates for different countries. Here in this example they are first making the qvd for exchange rates and then performing scheduled reload as usual...Ofcourse we require some credentials for accessing the data froma website but this problem is solved when you contact your IT/Networking team concerning this issue.

So what I feel that your administrator will be granting you certain rights to get the work done!

Extracting data from webpage

gauravkhare — Thu, 14 Apr 2011 10:09:24 GMT

The best way being that you first extract data and make a qvd of it...and then perform weekly/monthly etc reload, it is because what I have seen that in many cases it takes many hours before a qvd is finally made while accessing data from a website!

So make a qvd first..................!

In this way internet authentication over the server can be given during different time periods beacuse some companies don't allow internet access over the production server..So in this case the IT Team grants special rights for extracting data from the website!!

Extracting data from webpage

Thu, 14 Apr 2011 10:55:22 GMT

Thanks stephen and Gaurav,

i was trying to figure out the solution from the post

http://community.qlik.com/forums/t/38794.aspx

but not able.

Extracting data from webpage

Thu, 14 Apr 2011 11:14:05 GMT

Hi,

Find cURL and WGET on Google, Wikipedia etc . Here is some sample usage: http://community.qlik.com/forums/p/44155/175663.aspx#175663

-Alex

Re: Extracting data from webpage

stevedark — Fri, 06 May 2011 21:59:09 GMT

Hi there,

I've also run into the problem of getting data from a website that is protected with user credentials.

However, if anyone is looking to scrape data from a page where this isn't an issue I have created this short video:

Regards,

Steve

Extracting data from webpage

Sat, 27 Aug 2011 22:31:39 GMT

Steve,

thanks for posting that video!!

It helped me greatly with a project that requires data extraction from websites to be added to an existing app.

Extracting data from webpage

stevedark — Sun, 28 Aug 2011 07:36:39 GMT

Glad to be able to help! Cheers.

Re: Extracting data from webpage

alec1982 — Tue, 17 Jul 2012 14:09:34 GMT

I tried to follow the same way but I get 403 Forbidden as data only.

not sure if that because of the credentials..

Anybody can help..Please

Thxs,

Re: Extracting data from webpage

stevedark — Tue, 17 Jul 2012 21:41:05 GMT

There certainly are issues with opening web-pages with security on them, as there is no way of passing credentials in when making the call.

Youl could perhaps build a wrapper page in IIS that pulls down the page after passing in credentials that is then not secured with a username and password - but is only accessable to the IP address that your QlikView app is being reloaded on. This is not a particularly direct route though.

There are a number of articles on QlikCommunity relating to this - and you may be able to find a solution in the forums.

Hope that helps.

Steve

Re: Extracting data from webpage

alec1982 — Thu, 26 Jul 2012 14:04:28 GMT

I have created a batch file to open IE then add the credential and log in.

I added the batch file as a supporting task to the publisher and run the application on a success of this task.

it seems to work just fine!.

Re: Extracting data from webpage

stevedark — Thu, 26 Jul 2012 14:40:18 GMT

Nice! QlikView is obviously using part of the IE engine when it makes the call. Publisher is then carrying out all of its tasks in the same process. It should be possible to do the IE part in a QlikView macro and then trigger the reload from the same macro - so the same functionality could be gained without Publisher.

Could you perhaps post up the batch file, or the code from in the batch file - with the credentials removed, of course.

Regards,

Steve

Re: Extracting data from webpage

qlikviewforum — Tue, 28 Aug 2012 06:46:50 GMT

Given link is not working!

Re: Extracting data from webpage

qlikviewforum — Tue, 28 Aug 2012 06:47:00 GMT

Given link is not working!

Re: Extracting data from webpage

qlikviewforum — Tue, 28 Aug 2012 06:59:38 GMT

Hi Alec

Could you post up the batch file, or the code from in the batch file - with the credentials removed, of course. It will help us as I am working on the same.

Thanks,

qlikviewforum

Re: Extracting data from webpage

qlikviewforum — Tue, 28 Aug 2012 07:01:34 GMT

Thanks for the nice video. But in our case the website/sharepoint needs authentication to view the page. Please share the video if the same is possible.

Re: Extracting data from webpage

subin6600 — Tue, 28 Aug 2012 07:06:35 GMT

Hi Alec1982,

Could you please attach the batch file? That would be helpful for many of us here struggling to get the data read into Qlikview from sharepoint sites.

Regards,

Subin Thomas

Re: Extracting data from webpage

qlikviewforum — Wed, 29 Aug 2012 12:38:20 GMT

Hi Alec

Please share the batch file with us!

Thanks,

qlikviewforum

Re: Extracting data from webpage

qlikviewforum — Fri, 31 Aug 2012 12:13:32 GMT

Hi All,

I tried my level best but unfortunately couldn't able to succeed yet.

I read some where about QVSource. Is it possible to load the data from secured sharepoint site using the same? Any help please?

Thanks,

qlikviewforum