5 Replies Latest reply: May 5, 2017 6:58 AM by Marcos Paixao RSS

    URL Pagination using Qlik Sense Web file connector

    Cameron Scott

      I am trying to establish a connection to a website to pull in table data. The data spans several pages (displays 50 records at a time). The site uses a paramaterized URL so you can control the records displayed via the URL. Ideally I can have one web file connection that can loop through the pages itself. The only solution I have been able to come up with so far is to create a new web file connection for each parameterized URL, which is definitely not an elegant solution.

       

      This is what the URL structure is like.

      http://<URL>?yearId=2017&pageId=50

        • Re: URL Pagination using Qlik Sense Web file connector
          Marcos Paixao

          Dear Cameron,

           

          I solved this using one Web File. Here's a sample you can try.

          Capture-16.png

          Best Regards,

           

          Marcos

            • Re: URL Pagination using Qlik Sense Web file connector
              Cameron Scott

              Hi Marcos,

               

              Thanks for the sample code. I managed to get it to pseudo-work. I had to setup a web file connector and use the Lib:// rather than http://. This method with your code modified to fit my scenario did have the data load 'paging' sort-of. With this in place it would loop through the FOR EACH function for all the parameters passed to the variable and bring in a table of data. However, it was not actually paging through the URLs. Instead I would continually load the first page and add that to the table. So what I ended up with was a table of the first 50 records repeated x times (x being the number of parameter passed to the variable in the FOR EACH clause).

               

              To ensure the variable contained the correct value I added the variable to the table. The result showed that the FOR EACH clause was definitely looping through the parameters but for whatever reason the resultant URL would only load the first page over and over.

               

              If I try to manually load the different pages (i.e. change the &pageid in a browser to 0, or 50, or 100) the browser the different pages. I find this quite strange.

               

              Here is the code I am using (the text is pasted below if needed);

               

              This is the URL in the [lib:// ] connection;

              http://games.espn.com/fhl/playerrater?leagueId=27827&splitTypeId=0&seasonId=2017&startIndex=$(Pagination)

               

              With this code what I end up with is a list of the first 50 players repeated 21 times. Any thoughts?

               

              Note: I tried adding single quotes around the variable parameters and that did not make any difference.

               

              //** Sample Code **//

              FOR EACH Pagination IN 0,50,100,150,200,250,300,350,400,450,500,550,600,650,700,750,800,850,900,950,1000

               

              PlayerRaterRaw:

              LOAD

                  PLAYERS as Rank,

                  PLAYERS1 as Player,

                  STATUS as Owner,

                  "PLAYER RATER" as Goals,

                  "PLAYER RATER1" as Assists,

                  "PLAYER RATER2" as PlusMinus,

                  "PLAYER RATER3" as PIM,

                  "PLAYER RATER4" as SOG,

                  "PLAYER RATER5" as HIT,

                  "PLAYER RATER6" as BLK,

                  "PLAYER RATER7" as STP,

                  "PLAYER RATER8" as Wins,

                  "PLAYER RATER9" as Saves,

                  "PLAYER RATER10" as Shutouts,

                  "PLAYER RATER11" as GAA,

                  "PLAYER RATER12" as Save%,

                  "PLAYER RATER14" as Rating,

                  '2017' as Year,

                  $(Pagination) AS PageNumber

              FROM [lib://ESPN Player Rater - Paging]

              (html, codepage is 1252, embedded labels, table is @1);

               

              NEXT Pagination;