Save Website URL to a Field

[ciaran_mcgowan]

Hi all,

Strange one. I am scraping data from a website using the following URL format: www.website.com/data/1

I am incrementally increasing the 1, looping through about 100 pages. The problem is that while the above URL works for scraping the data, it is not the full URL - that is usually something like www.website.com/data/1-Title_of_page

Is there any way to return the full URL Qlikview is reading from? I need everything after the first number to use as the title for the row and to link it to the rest of my data.

UPDATE: I should point out that I need to return the full URL in the script as a field

[Anil_Babu_Samineni]

I don;t know where you want to add URL

Finaly, I suggest you to full URL like

Dim & '<URL>www.fullurl.com'

Thanks,

Anil

[ciaran_mcgowan]

I want to return the full URL in the script as a field. Then I'll be able to use that field to join my table to others using the URL as a key.

[Anil_Babu_Samineni]

Load * INLINE[

Field

Company

FROM

...

FullURL:

Load Field,

        Field & '<URL>www.fulladdress' as Field

Resident Directory;

Drop Table Directory;

Then you can use that full URL into that?

- Anil

[ciaran_mcgowan]

No. You misunderstand. I don't have the full URL, I only have the first part (which is enough for scraping). I am trying to find and return the full URL using a command similar to FilePath().

[Peter_Cammaert]

Did you try with the filexxxx() functions? FileName() for example?

[ciaran_mcgowan]

Yeah. Tried all of them but all they return is what's in the source location (i.e. text after the FROM statement) so all I'm getting are variations of www.website.com/data/1

[Peter_Cammaert]

Then this is probably a limitation of the FROM data source interface that doesn't know about the URL rewrite module that most web sites use since a long time. After all, it originated as a file interface...

Peter

[ciaran_mcgowan]

I figured that but was holding out for some hidden function that I hadn't heard of before 😕

[Peter_Cammaert]

It may be that some Qlik (formerly QVSource) Connector is able to return this meta-data to your script, as they were designed from the ground up to be web connectors. Not sure though.

Peter