TFTPfileExist, tloop, tinfiniteloop and tsleep process

[AAA999]

I need to check if the file exists on ftp server I am using tftpfileExist.
If the file is not found after certain period of time it should send a notification and loop again to check if the file is present. (Check every 1 minute). Once the file is found the job should kick and on completion it should send a notification and loop back to check for the next file.

current design below works fine

tftpconnection---onsubjob--> tFTPFileexist---onsubjob-->tFTPGet1---onsubjob-->TFileExist---onsubjob----Process---notification email ...
|flase
email notification

I am struggling which component to use tloop or tinfiniteloop and tsleep after which component and what variables i need to include. Can this be achived in one job? Would be helpful is some one could provide an example.

[Anonymous]

Hi
You need use a tLoop with while model to do the loop and stop the loop once the file is found, there is a global variable ((Boolean)globalMap.get("tFTPFileExist_1_EXISTS")) that can be used as condition of runIf connector, this will trigger other process based on the value of global variable.
Use a tsleep to pause N seconds.

Regards
Shong

[AAA999]

Thanks,

I tried the option not sure what I am missing. 

1) In case the file is already present  it sends an email

2) If file is not present it keeps exec in loop, later if the file is present it still keeps executing in loop and doesn't execute the main job. I have gone through forum but there is no good example for how to kick the main job and loop back once the job is executed for the next day

If conditionBoolean)globalMap.get("tFTPFileExist_2_EXISTS") == false

tloop (used while condition)

tsleep== 5 sec

what do you mean global variable value and how to define that.

I have attached a sample job.

.

tloop map.JPG

[AAA999]

Can anyone please provide a solution would be helpful.

Thanks

[Anonymous]

quick question: how many times do you want the job to re-check the file if it is not present? You need to set a number and exits the loop after retrying several times, otherwise, the job will execute the loop all the time if the file is not present.

[AAA999]

Want to check the file every 1 minute for 15 minutes. If the file is not present after 15 minutes then only send the email. In my current scenario it keeps sending email every 1 minute if the file is not present. The process should stop once the file has been processed and loop again next day. Presently I am using For condition in tloop and not passing any parameters in trun condition.

loop1.JPG
ftpchild.JPG
email_condition.JPG

[AAA999]

Can anyone please provide a solution?

[Anonymous]

First of all, I think a Talend Job is wrong for this requirement. Talend Jobs are batch. They start, do a job and end. If you want something to be continually polling for a file, it might be better to use Talend ESB. However, since you are close to your requirement here, I'll start with a requirement to ....

1. Poll for a file for 15 mins

2. If the file is found, some action takes place

3. If the file is not found after 15 minutes, send an email

The tLoop with the tRunJob and tSleep at the end looks like a good start. The adjustments I would make are these.....

1. The job run by the tRunJob needs to check for a file and if the file is found, do whatever task is required THEN submit a success response using a tBufferOutput (check the Talend documentation for how to use that). If the file is not found, it needs to send a fail status using the tBufferOutput.

2. After the tRunJob, you need to test the result of the tBufferOutput (which returns a data row from the tRunJob). This needs to set a variable. Maybe a globalMap variable. This variable needs to be used in your loop condition. If the file is found, the loop should end as the task is complete. Otherwise, the loop should continue for 15 minutes. Assuming the tSleep is for 1 minute, that is 15 loops.
3. Use a SubJobOk link to go from the tLoop to a tjava. From the tJava use RunIf links to decide whether to send your email. If the file was found (your variable set in step 2 can tell you this) send the email. Otherwise do nothing.

[AAA999]

The job run by the tRunJob needs to check for a file and if the file is found, do whatever task is required THEN submit a success response using a tBufferOutput (check the Talend documentation for how to use that). If the file is not found, it needs to send a fail status using the tBufferOutput.

I tried the tBufferOutput and went through the document but I couldn't figure out after which component to add the  tBufferOutput and how to send  a fail status using the tBufferOutput. Also in the tRunJob I am not passing any variable (what values need to be passed). In the job I created I am adding a "run if" condition for file not exist link from tFTPFileExist which I believe I should remove as it keep sending an email after every minute. For success the email component is in the end. 

2. After the tRunJob, you need to test the result of the tBufferOutput (which returns a data row from the tRunJob). This needs to set a variable. Maybe a globalMap variable. This variable needs to be used in your loop condition. If the file is found, the loop should end as the task is complete. Otherwise, the loop should continue for 15 minutes. Assuming the tSleep is for 1 minute, that is 15 loops.

Returns a data row from the tRunJob (how to pass this). the loop should continue for 15 minutes. Assuming the tSleep is for 1 minute, that is 15 loops.?

3. Use a SubJobOk link to go from the tLoop to a tjava. From the tJava use RunIf links to decide whether to send your email. If the file was found (your variable set in step 2 can tell you this) send the email. Otherwise do nothing.

tloop --> tJava (parent job?)

I am attaching the job I created. Please advice what i am missing.

Thanks

tchild.JPG
tWaitforfile loop.JPG

[Anonymous]

OK, I am a little stuck following your job design here. A best practice for writing jobs is to have the components ordered on your job in the same way words are ordered on a page. Top to bottom and left to right. Your subjobs (continuous coloured areas or components joined using OnComponentOk links) should be listed with the first subjob to run at the top and they should be connected via an OnSubJobOK link from first component of a subjob to first component of the next subjob. Like this....

This is just an example of the flow of components. This is just a job I have been working on.

With regard to your job you need to separate the checking for files and processing found files from the sending of the email. The sending of the email should be in the outer tLoop job. The child job  should be handling the checking for the file and the processing of found files OR identifying that the file has not been found. Your  child job needs to have two flows starting with checking that the files exists. If it does, then you process the file. If it does not, then you use something like a tFixedFlowInput component to send a failure message to a tBufferOutput. This way, the outer job can check the tRunJob after every loop. If the tRunJob has a failure message 15 times, then you stop the loop and send the mail.