topic Re: Guidance on using REST 'Next Token' to connect to Commerce7 in Connectivity & Data Prep

Guidance on using REST 'Next Token' to connect to Commerce7

bobiloco — Thu, 14 Dec 2023 17:21:49 GMT

I've been trying to set up a REST connector to pull all the values from the Commerce7 Orders API.

I'm not sure if I need to implement a loop or if this should work with just the connection details. Right now I keep pulling the same number of rows each time.

From their documentation, I should use cursor=start for the first loop and then use the cursor that is returned from the query.

Any help or examples that you could pass my way would be very helpful!

If I don't put "cursor = start " in the query parameters the select data screen shows Total instead of cursor and only pulls back one page.

Re: Guidance on using REST 'Next Token' to connect to Commerce7

Daniel_Pilla — Thu, 14 Dec 2023 17:58:34 GMT

Hi @bobiloco ,

If you cannot get it working through the REST Connector's GUI, you can set it to `cursor=start` and then insert the script. Once inserted, you can then add the `WITH CONNECTION` clause to the end and modify it, while putting the entire bit into a loop. This way, each time it loops, you use the `peek()` function to grab the current cursor and then modify the url (or header) to the next cursor -- continuing until either there is no more cursor or it is the same as the one prior (depends on the API and how they handle things). While this is admittedly more complex, it is extremely powerful and will prove a valuable feature to learn. You can see an example on how to use `WITH CONNECTION` here and here.

Re: Guidance on using REST 'Next Token' to connect to Commerce7

bobiloco — Thu, 14 Dec 2023 18:13:10 GMT

@Daniel_Pilla I suspected that this was the way forward and was starting to play with a second connection with different settings. I'll mark this as the answer after I play around a bit! Thank you!

Re: Guidance on using REST 'Next Token' to connect to Commerce7

bobiloco — Thu, 14 Dec 2023 22:17:52 GMT

This pointed me in the right direction. In the end I set it to custom pagination and put the cusor in the URL and it is now looping through and pulling back all records.

WITH CONNECTION( URL "https://api.commerce7.com/v1/order?cursor=$(vCursor)",