[resolved] Skip duplicate file to process

[Anonymous]

Hi
I have one talend job.
First, sub job process all the files in tfilelist folder and reject all files having not required records or format. And convert that file into .csv.bad extension.
After this main job starts working, and process all the files not converted to .csv.bad files. And convert those files into .csv.read.
My requirement is, if tfilelist receive and file which is already processed and converted to .csv.read, skip that file to process.
Any help ?

[Anonymous]

Hi,
Can you display screens from your job please ?
Thanks

[Anonymous]

Hi arnaudbroens
Here is my job
prntscr(dot)com/969t12

[Anonymous]

Did you set the file extension in the tFileList into "*.csv"?

[Anonymous]

You have many options on tFileList. Check your options and specially extensions ( says junmilsso ).
For extensions, if you select 'case sensitive : true', write "*.csv" and "*.CSV"

[Anonymous]

Yes I did
Files comes up with dates
For Example: XYZ_25122015.csv
If file is ok then that would be converted into XYZ_25122015.csv.read
But if file is not ok then that would be converted into XYZ_25122015.csv.bad without entering any single record into database.
My question is, if any file come up with same date which already has been  processed and converted into  XYZ_25122015.csv.read should be rejected.
In both tfilelist, I have set the mask like this "*.csv"
Any suggestion, how it can be done ?

[Anonymous · Accepted solution]

To do that, after your tFileList, iterate to a tFileExist with current filename suffixed to ".read". 
If exist rejected else continue ...

[Anonymous]

Hey arnaudbroens
Job is done now, thanks for reminding me tfileexist component.
Its been long time that I didn't use that component.
Best !!