topic Re: Save Website URL to a Field in QlikView

Save Website URL to a Field

ciaran_mcgowan — Thu, 30 Jun 2016 13:29:48 GMT

Hi all,

Strange one. I am scraping data from a website using the following URL format: www.website.com/data/1

I am incrementally increasing the 1, looping through about 100 pages. The problem is that while the above URL works for scraping the data, it is not the full URL - that is usually something like www.website.com/data/1-Title_of_page

Is there any way to return the full URL Qlikview is reading from? I need everything after the first number to use as the title for the row and to link it to the rest of my data.

UPDATE: I should point out that I need to return the full URL in the script as a field

Re: Save Website URL to a Field

Anil_Babu_Samineni — Thu, 30 Jun 2016 14:11:16 GMT

I don;t know where you want to add URL

Finaly, I suggest you to full URL like

Dim & '<URL>www.fullurl.com'

Thanks,

Anil

Re: Save Website URL to a Field

ciaran_mcgowan — Thu, 30 Jun 2016 14:16:26 GMT

I want to return the full URL in the script as a field. Then I'll be able to use that field to join my table to others using the URL as a key.

Re: Save Website URL to a Field

Anil_Babu_Samineni — Thu, 30 Jun 2016 14:19:34 GMT

Load * INLINE[

Field

Company

FROM

...

FullURL:

Load Field,

Field & '<URL>www.fulladdress' as Field

Resident Directory;

Drop Table Directory;

Then you can use that full URL into that?

- Anil

Re: Save Website URL to a Field

ciaran_mcgowan — Thu, 30 Jun 2016 14:35:57 GMT

No. You misunderstand. I don't have the full URL, I only have the first part (which is enough for scraping). I am trying to find and return the full URL using a command similar to FilePath().

Re: Save Website URL to a Field

Peter_Cammaert — Thu, 30 Jun 2016 14:37:34 GMT

Did you try with the filexxxx() functions? FileName() for example?

Re: Save Website URL to a Field

ciaran_mcgowan — Thu, 30 Jun 2016 14:53:26 GMT

Yeah. Tried all of them but all they return is what's in the source location (i.e. text after the FROM statement) so all I'm getting are variations of www.website.com/data/1

Re: Save Website URL to a Field

Peter_Cammaert — Thu, 30 Jun 2016 15:21:42 GMT

Then this is probably a limitation of the FROM data source interface that doesn't know about the URL rewrite module that most web sites use since a long time. After all, it originated as a file interface...

Peter

Re: Save Website URL to a Field

ciaran_mcgowan — Thu, 30 Jun 2016 15:24:23 GMT

I figured that but was holding out for some hidden function that I hadn't heard of before 😕

Re: Save Website URL to a Field

Peter_Cammaert — Thu, 30 Jun 2016 15:28:36 GMT

It may be that some Qlik (formerly QVSource) Connector is able to return this meta-data to your script, as they were designed from the ground up to be web connectors. Not sure though.

Peter