Data duplication in a flow with tHashInput and Tmap

[LDPtechAFB]

Hello,

i have parsed my data and stored intHashOutput,

Then i connect the job to the database and with a OnsubjobOk activate the next subjob who allows me to retreive data from my output thanks to the THashInput.

the problem is that datas are duplicated :

i don't know why the process loop and never stop implementing my database.

I configure the tmap with the option :

-Load once

Maybe the format of my database is an issue, i have some problem concerning the foreign key attribution. But the problem seems to come from the thashinput.

thank for your answers

Regards.

Lucas

[manodwhb]

@not specified not specified​ ,when you say data duplicating means ,is that the same data loading all three target tables? If yes then the problem may be with tMap configuration issue. can you share the based on what criteria are you loading to target tables from tMap.

[LDPtechAFB]

Yes the same datas are loaded to the database.

what didyou mean by the criteria ?

here is some configuration of my thashinput, tmap, thashoutput.

[JohnRMK · Accepted solution]

Hello ,

If there is no join between the inputs in your tMap, it will perform a scalar product (produit scalaire) so all the data will be duplicate.

You can use tUniqRow on the output to deduplicate

[LDPtechAFB]

Yes it seems to be that,

But i cannot join the input to each other because there is no similarities between these datas.

I tried with a tUniqRow, but it doesn't work.

I can only put link in my output of the tmap but it seems to be impossible (foreign key of the DB). So i don't know what to do.

If you have some clues !

[manodwhb]

@not specified not specified​ , Have you pointed all tHashInputs to tHsashOutput_4 only ? then that is problem.

[manodwhb]

@not specified not specified​ ,how many columns do you have in each tHashInput's and data types ?

[LDPtechAFB]

26, 33, 70 for number of column, with data types String, interger, Date, boolean, float with automatic conversion of type.

By the way, the method with the tUniqRow actually work (i just use the wrong configuration of the component).

Now i have a problem from my input

java.lang.NumberFormatException: For input string: ""

the tUniqrow may work now. But for now, i have to fix this issue

[manodwhb]

@not specified not specified​ , what is the join key if you do not have join column the it will take cartesian join. and the output section how are you populating to tDBOutputs. if you have are taking the cartisian join and populating as same then you will have same data across the all target tables.

[LDPtechAFB]

I created a column for each input with a default value and i join these column together to avoid the cartesian join. it seems to work well, i will see in the future. Thanks for your answers