topic Incremental or Delta load in Talend Studio

Incremental or Delta load

BA621858188 — Fri, 15 Nov 2024 22:51:09 GMT

Hi All,

I have gone through videos which are based on Incremental data load where the source and target both are DB.

But I have source as database and target as File. This is insert/Update both operations based on timestamp field.

Could any one help me here.

Thanks.

Re: Incremental or Delta load

Anonymous — Thu, 09 Jun 2022 00:07:03 GMT

If you are wanting to update a file you will need to read all of the data into memory within the job (maybe using the hsqldb components https://help.talend.com/r/en-US/8.0/hsqldb/hsqldb), compare that against the data in your db, then create a new/updated dataset in memory and write it to a new file.

Re: Incremental or Delta load

BA621858188 — Thu, 09 Jun 2022 16:57:06 GMT

@Richard Hall I can fetch the increment records based on query (using date column) but how can I compare it for the next load? (One way is to store the value in context parameter).

The one difference here in my scenario that I need to check increment query using separate as I am not going to fetch any cols from that table and I am going to fetch the appropriate cols from separate table.

Do you have any job design to get more insights?

Re: Incremental or Delta load

Anonymous — Thu, 09 Jun 2022 21:56:33 GMT

Your max date stamp in your target should be used as your watermark if I am understanding your requirement. So when you query the file's data, you will need to find the max date. Once you have the max date, you can use that to query your source data (in the db) for all records greater than the file's max date.

Re: Incremental or Delta load

BA621858188 — Sun, 26 Jun 2022 15:44:05 GMT

@Richard Hall Can I use Insert update flows same as SCD TYPE2 in incremental/delta load flow.? I need to load the data each day to new file.