tFileList : how to keep the job running in spite of files which can't be read and how to identiy files which can't be read

[BeginnerNeedsHelpPlease]

Hello

I have many files in a folder and I need to read all of them.

Actually i need to read in a specific woksheet.

The issue is that some files doesn't have the woksheet I have to read.

And there are too many files, I can't open each of them and check every one of them.

I used a tFileList and I specify the folder and the worksheet.

I run the job and my issus is that when the file doesn't have the worksheet, the job stops and I have to remove the file which made the job stop and restart the job.

It is a waste of time.

Is it possible to make the job run until the end, even if it read some bad files?

I'd like to create a main job where I read all the files.

In the meantime, I have an other job which is triggered on an error by the main job and this other job will store in a file all the filenames which doesn't have the worksheet and which are supposed to trigger the error.

if you have another solution I'm also interested.

thank you

[gjeremy1617088143]

Hi could you send a picture of the configuration of your tFileList ?

I think first you have to list all the files then filter on those who have worksheet.

Send me Love and Kudos

[Prakhar1]

Hello,

for this type of scenarios I will suggest you to use the master-child job concept.

Your master job will look like this

tFileList -->iterate-->tRunJob

In tRunJob uncheck the "Die on Run job" option.

Now your child job will have the logic which you are using to read the sheet from the excel file.

Also, I would suggest you the move the file to other folder when it is successful, so in the source directory only those files be present which are failed.

[BeginnerNeedsHelpPlease]

can you make a screenshot of this configuration please?

I'm confused with master-child job concept

here is a screenshot of what i tried to do

 

here are some example of files with my test

 

thank you

[Prakhar1]

I can not provide you template for exact job design .

you can look into below link for reference

https://help.talend.com/viewer/attachment/wDRBNUuxk629sNcI0dNYaA/ISv4ROBBOhDrjPmXeswh1w

I can explain you here

1) create two jobs

2) in the first job

tFilelist --> iterate --> tRunJob(component)

3) Create one more job with the logic

tFileInputExcel --> tUnite-->tFileoutput

||

onComponentError

||

tFileCopy (now move the file to other directory if the file has error )

4) Now in the first job you have to call this job in tRunJob component (you can refer the link for this)

5) also you have to create one context in second job to pass the file name value from master job to child job

If this all not help then I will try to create a template for you.😅

[BeginnerNeedsHelpPlease]

let me try this week-end.

[BeginnerNeedsHelpPlease]

hello again,

I followed your explanation but I coulnd't do it:

I created 2 jobs:

A1_job1

A2_job2

in A1, I have the tRunJob

 

in A2, i have the job according with described what I want to to do.

I uncheck the "Die on Run job" option. It looks like I can't add a context. Also I don't know what other options I have to select

For A2, I just defined what I was trying to do. 

I need to add another job so that I i removed the bad files but I'm stuck with the first steps.

The files I want to read are easy. Let's says I have 4 files and one is bad and can't be read :

 

can you tell me what is wrong in what i'm doing?

[Prakhar1]

Ok you prepared right job but missed somethings..

1) in your A2 job, there will be no tFileList, because you are using tFileList in you A1 job, so remove it from the A2.

2) Add context first in A2 then you can see it in A1 job.

3) Now when you add then context in A2 job , go to tRun in A1 and add again you will see the same context which you created in A2. Now in the value of that context write this == (String)globalMap.get("tFileList_1_Current_File_Path")

4) And in A2 job, in tFileInputExcel, in the File Path write the context name you created in A2 job.

Let me know if this works for you or not.

[BeginnerNeedsHelpPlease]

Hello,

I need some little help please. I followed the 4 steps, I'm confused with step 2

step 1 : OK

step 2 : I'm confused. How can I add a context?

step 3 : I think I understand but I can't try it because I don't know what to do in step 2

step 4 : OK I understand but I can't try it because I don't know what to do in step 2

thanks for the help

[Prakhar1]

1) Go to Context tab , as shown in the below image 

2) On this tab you can see option to add context , so click on "+" icon and add a context "FilePath" of type "String" just like the below image.

 

3) Now write the same context in the "tFileInputExcel" in the filepath field, just like the below image