There is a requirement to connect to an API and fetch data for 40k products.
For which, we have built a loop to fetch 40k products data from API. But it fails while ruining for 40k times to hit the endpoint and looking for the data.However, the same script works for 2000 products
.
What is the best methodology and way to automate this? And can fetch data for all 40k products in one run.
LIB CONNECT TO 'API ';
[List of Products]:
LOAD
SectorId as SectorId,
LocaleId as LocaleId,
ProductId as ProductId ,
CatalogID as CatalogID
FROM [ProductIds.xlsx]
(ooxml, embedded labels, table is Sheet1);
let vNumofRowsProduct = NoOfRows('Product')-1;
for i = 0 to $(vNumofRowsProduct)
let vProductId = peek('ProductId',$(i),'Product');
let vSectorId = peek('SectorId',$(i),'Product');
let vLocaleId = peek('LocaleId',$(i),'Product');
Let VCatalogID = peek('CatalogID',$(i),'Product');
set ErrorMode=0;
RestConnectorMasterTable:
SQL Select
FROM JSON (wrap on) "root"
with connection (url "https://URL/ vLocaleId)/$(VCatalogID)/products/$(vProductId).summary")
;
Trace 'Trace'&'---'&$(vProductId);
if i=0 then
ProductDetail:
Load
Filed here
Resident RestConnectorMasterTable;
else
concatenate(ProductDetail)
Load
Field here
Resident RestConnectorMasterTable;
EndIf
drop table RestConnectorMasterTable ;
next i ;
drop Table Product;
