topic Re: [resolved] Skip duplicate file to process in Talend Studio

[resolved] Skip duplicate file to process

Anonymous — Mon, 23 Nov 2015 08:54:38 GMT

Hi
I have one talend job.
First, sub job process all the files in tfilelist folder and reject all files having not required records or format. And convert that file into .csv.bad extension.
After this main job starts working, and process all the files not converted to .csv.bad files. And convert those files into .csv.read.
My requirement is, if tfilelist receive and file which is already processed and converted to .csv.read, skip that file to process.
Any help ?

Re: [resolved] Skip duplicate file to process

Anonymous — Mon, 23 Nov 2015 16:08:22 GMT

Hi,
Can you display screens from your job please ?
Thanks

Re: [resolved] Skip duplicate file to process

Anonymous — Tue, 24 Nov 2015 06:49:23 GMT

Hi arnaudbroens
Here is my job
prntscr(dot)com/969t12

Re: [resolved] Skip duplicate file to process

Anonymous — Tue, 24 Nov 2015 07:16:33 GMT

Did you set the file extension in the tFileList into "*.csv"?

Re: [resolved] Skip duplicate file to process

Anonymous — Tue, 24 Nov 2015 09:50:19 GMT

You have many options on tFileList. Check your options and specially extensions ( says junmilsso ).
For extensions, if you select 'case sensitive : true', write "*.csv" and "*.CSV"

Re: [resolved] Skip duplicate file to process

Anonymous — Tue, 24 Nov 2015 13:02:42 GMT

Yes I did
Files comes up with dates
For Example: XYZ_25122015.csv
If file is ok then that would be converted into XYZ_25122015.csv.read
But if file is not ok then that would be converted into XYZ_25122015.csv.bad without entering any single record into database.
My question is, if any file come up with same date which already has been processed and converted into XYZ_25122015.csv.read should be rejected.
In both tfilelist, I have set the mask like this "*.csv"
Any suggestion, how it can be done ?

Re: [resolved] Skip duplicate file to process

Anonymous — Tue, 24 Nov 2015 13:30:01 GMT

To do that, after your tFileList, iterate to a tFileExist with current filename suffixed to ".read".
If exist rejected else continue ...

Re: [resolved] Skip duplicate file to process

Anonymous — Wed, 25 Nov 2015 06:18:12 GMT

Hey arnaudbroens
Job is done now, thanks for reminding me tfileexist component.
Its been long time that I didn't use that component.
Best !!