Incremental or Delta load

[BA621858188]

Hi All,

I have gone through videos which are based on Incremental data load where the source and target both are DB.

But I have source as database and target as File. This is insert/Update both operations based on timestamp field.

Could any one help me here.

Thanks.

[Anonymous]

If you are wanting to update a file you will need to read all of the data into memory within the job (maybe using the hsqldb components https://help.talend.com/r/en-US/8.0/hsqldb/hsqldb), compare that against the data in your db, then create a new/updated dataset in memory and write it to a new file.

[BA621858188]

@Richard Hall​ I can fetch the increment records based on query (using date column) but how can I compare it for the next load? (One way is to store the value in context parameter).

The one difference here in my scenario that I need to check increment query using separate as I am not going to fetch any cols from that table and I am going to fetch the appropriate cols from separate table.

Do you have any job design to get more insights?

[Anonymous]

Your max date stamp in your target should be used as your watermark if I am understanding your requirement. So when you query the file's data, you will need to find the max date. Once you have the max date, you can use that to query your source data (in the db) for all records greater than the file's max date.

[BA621858188]

@Richard Hall​  Can I use Insert update flows same as SCD TYPE2 in incremental/delta load flow.? I need to load the data each day to new file.