topic Dataextraction from a Webpage in QlikView

Dataextraction from a Webpage

prieper — Wed, 15 Sep 2010 13:13:54 GMT

Dear Experts,

I am trying to extract data from a webpage, so far nothing special, but am not able to extract the data. The URL reads

http://en.chineseshipping.com.cn/html/ccfi.asp

The required data should reside in table @8, but QV extracts only the header and columns, but not the values itself (and neither the dates in the header). Could neither find something in the source-code of the page. Looks as if the data are somehow generated over the existing webpage - Has someone a solution for this?

Peter

Dataextraction from a Webpage

disqr_rm — Wed, 15 Sep 2010 15:39:30 GMT

Pretty tricky one. I know developers in China are very smart.

I think you need to do IE automation to open the page and then save it as a text file on your hard drive. Then in QV script you need to read the text file and parse out the stuff you want. I don't know why it doesn't show contents right when you save as html, but save as text shows the numbers you are looking for.

Something like this should help: http://www.experts-exchange.com/Programming/Languages/Visual_Basic/VB_Script/Q_22786372.html

Sorry, not a 100% solution, but may give you some ideas.

Rakesh

Dataextraction from a Webpage

Wed, 15 Sep 2010 16:39:20 GMT

Hi Peter, looks like the actual values that appear in the table are being loaded asynchronously (note the 'I'm doing something in the background' progress wheel that displays until the table's populated) once the web page itself has loaded, hence they don't appear in the html source.

You may want to google around "screen scraping ajax" for pointers on how to figure out how a given site's set up to grab data asynch... (almost certainly some javascript (the 'j' in ajax) in the mix somewhere).

Good luck, Jason

Dataextraction from a Webpage

prieper — Mon, 20 Sep 2010 10:01:30 GMT

Thanks Jason,

played around a bit, but still looks to be pretty cumbersome.

++ Peter