topic Re: Load in incremental mode with TOS for Data Integration in Talend Studio

Load in incremental mode with TOS for Data Integration

INESBK — Mon, 15 May 2017 15:05:13 GMT

Hi,

I read a large volume of files (which have the same format and the same columns) in order to insert them in sql server.
Each time I have to insert(load) new lines of files (each evening for example).

My question is: Can i use INSERT IF Not Exist(action in data) to do the trick or I have to create a CDC.

What is the best solution?
If the solution is CDC how can I create it?

Best regards.

Thanks in advance.

Re: Load in incremental mode with TOS for Data Integration

Anonymous — Tue, 16 May 2017 07:56:33 GMT

Hello,

Talend Changed Data Capture(CDC) feature will be available in talend subscription solution.

If you want to capture the changed data and only load these changed data into target table to achieve table sync, you can compare tables by using tMap.
The work flow should be:

Target DB-->tMap(make inner join on your input and set the "Catch lookup inner join reject" as true)-->output
Source DB-->
The output will be the changed data.

Let us know if it is Ok with you.

Best regard

Sabrina

Re: Load in incremental mode with TOS for Data Integration

INESBK — Tue, 16 May 2017 09:00:07 GMT

Could you please explain to me more.

My job is like this

tfile_list----->tSystem----->tExtractDelimetedFields----->tMap----->tMsSqlOutput_1------>tMsSqlOutput_2(for reject rows)

So i should to connet tMsSqlOutput_1 with an other tMap to consider it as a source?

Thanks.