How to parallelise a db input

[MattE]

Hi, i've got a fairly simple job (see below). I get some data out of redshift via sql, use that data to construct a json body string then pass that data to a tRunJob which essentially just does a REST API call in a loop. The problem is the sql brings back around 8000 rows a day and calling each one in sequence takes a number of hours so i want to parallelise the output of the tDBInput component.

I've tried to use the iterate row out of tDBInput but then i don't know how to reference the row data as you would usually do something like rownumber.fieldname but there is no row number for the iterate output. 

Thanks

[Shicong_Hong · Accepted solution]

Hi

You need a tFlowToIterate component to iterate input rows, then you can access the current column on other component like this:

rowNumber.columnName

(eg: row1.name)

Click Iterate connector and check the 'Enable parallel execution' checkbox in the basic setting panel if you want to parallelise the input rows.

Regards

Shicong

[MattE]

Thanks Shicong this worked.