topic Re: Remove files that as the same content in Talend Studio

Remove files that as the same content

Anonymous — Sat, 16 Nov 2024 06:53:40 GMT

Hello !

I'd like to remove all files that have the same content but keep one.

The final result should be files with all different content.

My file name format is fileName_timestamp.csv

For exemple :

My directory looks like this :

- fileName_t1m3st4mp.csv

- fileName_0th3rt1m3st4mp.csv

- fileName_4n0th3rt1m3st4mp.csv

Content in my files looks like this :

fileName_t1m3st4mp.csv

This is a content

fileName_0th3rt1m3st4mp.csv

This is a content

fileName_4n0th3rt1m3st4mp.csv

This is a different content

When i run the job :

fileName_0th3rt1m3st4mp.csv should be deleted

Now my directory should only have :

- fileName_0th3rt1m3st4mp.csv

- fileName_4n0th3rt1m3st4mp.csv

using Talend ESB 7

If you have any suggestion, please do !

Thanks !

Re: Remove files that as the same content

akumar2301 — Wed, 09 Jan 2019 16:16:08 GMT

Try with tFileList , tMemoriseRow tFileCompare and tFileDelete .

Not sure if these are part of ESB

Re: Remove files that as the same content

Anonymous — Thu, 10 Jan 2019 13:37:19 GMT

Thanks for your response !

Those components are indeed in ESB.

I need to compare each files with all the others, i'm not sure how i can do that with a FileCompare component since it only allow 1 input.

Can you guide me through your thinking ?

Best regards,

Re: Remove files that as the same content

akumar2301 — Thu, 10 Jan 2019 15:29:11 GMT

You are right with tFileCompare you might have some issues.

1) Actually you need to get the checksum of each file using

2) Find files having same checksum and delete the duplicate file.

tFileList --> tFileProperties(MD5 option) --> tFileOutput

onSubJobOK

tFileInput --> tUniqRow (getDuplicate filename based on checksum) --> tFlowtoInterate --> tFileDelete

This should work.

Re: Remove files that as the same content

Anonymous — Thu, 10 Jan 2019 15:45:56 GMT

Here you're mainly checking the file name not the actual content.

I think i found something. I can log content and filename independently but can't find a way the get both of them at the same time.

My goal here is get a output that contains all the file names and file content. (fileName;fileContent)

I guess i'll be able to use a tUniqRow to check duplicate content once i've figured out this.....

Re: Remove files that as the same content

akumar2301 — Thu, 10 Jan 2019 15:50:52 GMT

tFileProperties will get checksum based on Filecontent not filename.

Re: Remove files that as the same content

akumar2301 — Thu, 10 Jan 2019 16:09:50 GMT

it worked . Removed duplicate files. Try once.

Re: Remove files that as the same content

akumar2301 — Fri, 11 Jan 2019 14:24:20 GMT

did it solved your problem ?

Re: Remove files that as the same content

Anonymous — Tue, 15 Jan 2019 09:01:18 GMT

What's the component you renamed "selectMD5Option" ?
I'll try that