topic Re: Inserting just the new data in Talend Studio

Inserting just the new data

Anonymous — Wed, 26 Apr 2017 09:56:25 GMT

I want to know if there is any way to do the below process with talend.

evry time the job is runing, i want to insert just the new data in the source datatable .

Re: Inserting just the new data

TRF — Wed, 26 Apr 2017 13:08:06 GMT

It depends...

If you've a primiray key on these tables, insert will fail as soon as a value already exists (and only new rows will be inserted). But can be considered a dirty approach specially if you have a large number of rows in the source for only a few of new ones.

If you want to avoid rejection for existing rows, you need to join (inner join in a tMap) rows from the source with existing rows in the target (the lookup) and "catch lookup inner join reject" to identify only new rows to be inserted.

An other approach is to store the datetime of the last run somewhere (table or file) and use this information as a filter for the select query based on a column indicating the datetime the row was created is the source database.

Hope this helps,

Re: Inserting just the new data

INESBK — Tue, 16 May 2017 15:48:57 GMT

Hi,

I've connected the TMsSqlOutput with tmap as Lookup and connected Tmap to the same TMsSqlOutput but i've got this warnings (Aucun lien main n'est défini pour ce composant) and my job doesn't work

Could you help me @TRF ?

Re: Inserting just the new data

TRF — Tue, 16 May 2017 16:59:10 GMT

You cannot have a tMap without a main input flow.

Here, tMap_3 is connected with a lookup (row7) and an outpout flow (sortie).

Because the link connected to a tMap is automatically considered as the main input flow, I suppose you have deleted something which was connected to the tMap_3.

If you just want to make some transformations in tMap_3, row7 should be considered as the main input flow for tMap_3.

Cut row7, then reconnect the Output_Trend component to tMap_3.

Should solve your problem.

Bon courage.

Re: Inserting just the new data

INESBK — Tue, 16 May 2017 19:59:10 GMT

I need (lookup) to insert just new data, this is why tMap_3 is connected with a lookup (row7).

If i reconnect the Output_Trend component to tMap_3 i can't have Lookup.

Thanks for your help.