topic Re: Duplicate check between files while using tFilelist in Talend Studio

Duplicate check between files while using tFilelist

Anonymous — Sat, 16 Nov 2024 04:22:35 GMT

Hi Team,

I am trying to load files from a directory to MySql Output table
I used tFileList > tFileinputDelimited>tMap>tMySqlOutput design to iterate through the files
Now I want to remove duplicate data between files. ie, check the data based on a column or combination of 2-3 columns between the files
For example: if month column of first file contains data NOV and if the second file contains same month data as NOV, job should neglect the second file to load
Please help me to implement this concept in my job

Re: Duplicate check between files while using tFilelist

TRF — Tue, 15 Oct 2019 15:11:06 GMT

I suppose all your input files are based on the same schema. In such a case, you can read all the input files and push the result to a single temporary file the eliminate the duplicate records before to go into MySQL.

The design should look like this:

tFileList--(iterate)-->tFileInputDelimited-->tFileOutputDelimited(with Appen option ticked)

+(OnSubjobOK)

tFileInputDelimited-->tUniqRow-->tMysqlOutput

Re: Duplicate check between files while using tFilelist

Anonymous — Tue, 15 Oct 2019 15:24:38 GMT

Thanks TRF for providing the job design and concept. Can you please tell me how will I identify and remove the duplicates from the temporary file and distinguish the data is from from first file and second file to find out the correct data.

Re: Duplicate check between files while using tFilelist

TRF — Tue, 15 Oct 2019 15:52:32 GMT

You need a new field into the temporary file.

Change the design like this:

tFileList--(iterate)-->tFileInputDelimited-->tMap-->tFileOutputDelimited(with Appen option ticked)

+(OnSubjobOK)

tFileInputDelimited-->tUniqRow-->tMysqlOutput

In the tMap you add a field into the output flow (let say filename) and use this expression to populate this field:

((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))

Change "tFileList_1" depending on your component real name.

Is that what you expect?

Re: Duplicate check between files while using tFilelist

Anonymous — Wed, 16 Oct 2019 15:20:40 GMT

Thanks TRF, I have tried this approach and it is working as how the files are placed in the the directory.The order of the file in tFileList is from the last file in the directory to the first file, right? I mean the order of the files. Can we specify the order of file load in tFileList or using any component? Also How will I specify the filenames in tfileinputDelimited, tFileOutputDelimited in the main job and tFileinputDelimited in the subjob? using ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))?