Run Talend based on last Date/Time field in the database

thomas_2583 — Fri, 15 Nov 2024 22:48:23 GMT

Hi,

I have a table within the data warehouse which has a datetime field (upload_datetime)

As it stands, I have just done an insert of data so I have all of the data as 2022-06-20 20:30:00

This table has c. 500k records.

What I need to do is to update or insert on the next run, but don't want to have to process the full data again if at all possible. What I was hoping to do would be to just update or insert where the datetime field field is greater than that last datetime (i.e 2022-06-20 20:30:00)

Is that possible please, or if not, how can I better the performance of each refresh as this table will update every half an hour and to check back through that many records is too time consuming.

This is my job design: TDBInput ---------> TMap ---------> TDBOutput

Any help will be much appreciated.

Re: Run Talend based on last Date/Time field in the database

Anonymous — Mon, 04 Jul 2022 07:41:13 GMT

In the beginning of job, select the last datetime(i.e 2022-06-20 20:30:00) from target table and store the value to a context variable, this value will be used as filter condition for input data, provided that the input data also has a datetime field.

Regards

Shong

topic Re: Run Talend based on last Date/Time field in the database in Talend Studio

Run Talend based on last Date/Time field in the database

Re: Run Talend based on last Date/Time field in the database