topic How to parallelise a db input in Talend Studio

How to parallelise a db input

MattE — Thu, 04 Jul 2024 08:15:17 GMT

Hi, i've got a fairly simple job (see below). I get some data out of redshift via sql, use that data to construct a json body string then pass that data to a tRunJob which essentially just does a REST API call in a loop. The problem is the sql brings back around 8000 rows a day and calling each one in sequence takes a number of hours so i want to parallelise the output of the tDBInput component.

I've tried to use the iterate row out of tDBInput but then i don't know how to reference the row data as you would usually do something like rownumber.fieldname but there is no row number for the iterate output.

Thanks

Re: How to parallelise a db input

Shicong_Hong — Fri, 05 Jul 2024 07:46:10 GMT

You need a tFlowToIterate component to iterate input rows, then you can access the current column on other component like this:

rowNumber.columnName

(eg: row1.name)

Click Iterate connector and check the 'Enable parallel execution' checkbox in the basic setting panel if you want to parallelise the input rows.

Regards

Shicong

Re: How to parallelise a db input

MattE — Mon, 29 Jul 2024 09:16:59 GMT

Thanks Shicong this worked.