There is a requirement to connect to an API and fetch data for 40k products.
For which, we have built a loop to fetch 40k products data from API. But it fails while ruining for 40k times to hit the endpoint and looking for the data.However, the same script works for 2000 products
. What is the best methodology and way to automate this? And can fetch data for all 40k products in one run.
LIB CONNECT TO 'API ';
[List of Products]: LOAD SectorId as SectorId, LocaleId as LocaleId, ProductId as ProductId , CatalogID as CatalogID FROM [ProductIds.xlsx] (ooxml, embedded labels, table is Sheet1);
let vNumofRowsProduct = NoOfRows('Product')-1;
for i = 0 to $(vNumofRowsProduct)
let vProductId = peek('ProductId',$(i),'Product'); let vSectorId = peek('SectorId',$(i),'Product'); let vLocaleId = peek('LocaleId',$(i),'Product'); Let VCatalogID = peek('CatalogID',$(i),'Product');
set ErrorMode=0; RestConnectorMasterTable: SQL Select FROM JSON (wrap on) "root" with connection (url "https://URL/ vLocaleId)/$(VCatalogID)/products/$(vProductId).summary") ;
Trace 'Trace'&'---'&$(vProductId);
if i=0 then ProductDetail: Load Filed here Resident RestConnectorMasterTable; else concatenate(ProductDetail) Load Field here Resident RestConnectorMasterTable; EndIf drop table RestConnectorMasterTable ; next i ;
