Qlikview REST connector NextURL-pagination XML

[manuelaengl]

Hi everbody,

we try to load data from MS Sharepoint via REST Connector 1.2.0 into a QlikView application (11.20 SR12). Pagination on Server side is set to 100 and Sharepoint uses NextURL-Pagination. The XML looks pretty much like the following (only confidential data removed):

<?xml version="1.0" encoding="utf-8" ?>
- <feed xml:base="http://some.sharepoint.de/xy/_api/" xmlns="http://www.w3.org/2005/Atom">
   <id>b8ac15ff-dbc5-4b78-8a8d-33d8c31b8a50</id>
   <title />
   <updated>2017-12-14T14:11:27Z</updated>
- <entry m:etag=""22"">
   <id>Web/Lists(guid'608da7f5-bf0c-4426-92de-e5cdf3e4e7f2')/Items(500)</id>
   <category term="SP.Data.AufgabenListItem" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" /> 
   <link rel="edit" href="Web/Lists(guid'608da7f5-bf0c-4426-92de-e5cdf3e4e7f2')/Items(500)" />
   <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/FirstUniqueAncestorSecurableObject" type="application/atom+xml;type=entry" title="FirstUniqueAncestorSecurableObject" href="Web/Lists(guid'608da7f5-bf0c-4426-92de-e5cdf3e4e7f2')/Items(500)/FirstUniqueAncestorSecurableObject" /> 
   <title />
   <updated>2017-12-14T14:11:27Z</updated>
- <author>
  <name />
   </author>
- <content type="application/xml">
  - <m:properties>
     <d:FileSystemObjectType m:type="Edm.Int32">0</d:FileSystemObjectType>
     <d:Id m:type="Edm.Int32">500</d:Id>
     <d:Status>abgeschlossen</d:Status>
    </m:properties>
    </content>
   </entry>
  <link rel="next" href="http://some.sharepoint.de/xy/_api/web/Lists/getbyid('608DA7F5BF0C442692DEE5CDF3E4E7F2')/items?%24ski..." />
</feed>

We try to load with following settings:

• Pagination Type:Next URL
• Next URL field path: feed/link/attr:href (--> NextURL see line 24)
• Look in header: deactivated

Test Connection is okay and we can load data, but QlikView loads only first 100 entries from the initial URL. And then proceeds with the script without any error message.

Really hope someone can help!

Regards Manuela

[manuelaengl · Accepted solution]

Unfortunately, there is no real solution as it seems to be a bug in recognizing the XML-structure. But we found a workaround.

You can use custom pagination:

1. With a Loop you can iterate trough data packages. Starting with Initial URI

2. at the end of each select-statement you can write the next URI in a variable

3. Next.....

4. In last data package the NextHref ist empty --> Exit for

Best regards,

Manuela

[aferrari7]

Manuela, hi!

I used your script to paginate a .json page. Seem's to work fine in the beggining but it's not fetching all the records the MasterTable have to the table i need.

Coul you help me with that, please?

Best regards,

Alisson

[balabhaskarqlik]

Did you resolved the issue, please post the solution to help fellow people.

[manuelaengl]

Hey Alisson,

without the original data und code just can guess.. either your total is bigger than 6000 (you can increase or append with a little code first reading the total amount of records) or the pageSize is more than 100 (depends on source system).

Best regards,

Manuela

[manuelaengl]

Hey Bala,

for the solution to the initial problem pls see above. As already mentioned, it's only a workaround but in my case that was enough.

Best regards,

Manuela

[aferrari7]

Manuela,

thank you for sharing your LOOP, i'm using here and works perfectly!.

But firstly i had to preload symbols count to "No Limit" so that i can have the "Root" table to paginate through the .json.

Best Regards,

Alisson

[pawwy1415]

Hi https://community.qlik.com/t5/user/viewprofilepage/user-id/50273

We are also facing the same issue. But we are trying in the Qliksense. When we tried with "/Items?$Top=300000" in the URL its fetching records up to 21000 but still some dates are missing.

By removing Top=300000 by passing any field here fetching only 100 records.

Could you please let me know how to create the Custom connection in the qliksense. The  above qlikview  connection not working in Qliksense.