topic Iterate a job in Talend Studio

Iterate a job

dwhdeveloper — Sat, 16 Nov 2024 09:33:26 GMT

Hello,

I need to extract data from big query for last 2 years.

I populated a table with dates with last 2 years.

Now I want to read date from this table and get daily data from big query for last 2 years.

What would be the best way iterate this job?

PS. Table_date_range in big query failed because of too much data.

Re: Iterate a job

Anonymous — Fri, 07 Jul 2017 04:34:05 GMT

Hi
If I understand your request well, you need to iterate each date and pass it to the query as a parameter to query only the data based on the current date.
Take a look at the tFlowToIterate component documentation and learn this component. eg:
Read date from data source--main(row1)--tFlowToIterate--iterate-->tBigQueryInput....

on tBigQueryInput, you can access the current value of date_column with this expression:
(java.util.Date)globalMap.get("row1.date_column")

Regards
Shong