topic Re: TFTPfileExist, tloop, tinfiniteloop and tsleep process in Data Quality

TFTPfileExist, tloop, tinfiniteloop and tsleep process

AAA999 — Sat, 16 Nov 2024 04:02:28 GMT

I need to check if the file exists on ftp server I am using tftpfileExist.
If the file is not found after certain period of time it should send a notification and loop again to check if the file is present. (Check every 1 minute). Once the file is found the job should kick and on completion it should send a notification and loop back to check for the next file.

current design below works fine

tftpconnection---onsubjob--> tFTPFileexist---onsubjob-->tFTPGet1---onsubjob-->TFileExist---onsubjob----Process---notification email ...
|flase
email notification

I am struggling which component to use tloop or tinfiniteloop and tsleep after which component and what variables i need to include. Can this be achived in one job? Would be helpful is some one could provide an example.

Re: TFTPfileExist, tloop, tinfiniteloop and tsleep process

Anonymous — Mon, 25 Nov 2019 08:22:46 GMT

Hi
You need use a tLoop with while model to do the loop and stop the loop once the file is found, there is a global variable ((Boolean)globalMap.get("tFTPFileExist_1_EXISTS")) that can be used as condition of runIf connector, this will trigger other process based on the value of global variable.
Use a tsleep to pause N seconds.

Regards
Shong

Re: TFTPfileExist, tloop, tinfiniteloop and tsleep process

AAA999 — Mon, 25 Nov 2019 16:16:28 GMT

Thanks,

I tried the option not sure what I am missing.

1) In case the file is already present it sends an email

2) If file is not present it keeps exec in loop, later if the file is present it still keeps executing in loop and doesn't execute the main job. I have gone through forum but there is no good example for how to kick the main job and loop back once the job is executed for the next day

If conditionBoolean)globalMap.get("tFTPFileExist_2_EXISTS") == false

tloop (used while condition)

tsleep== 5 sec

what do you mean global variable value and how to define that.

I have attached a sample job.

tloop map.JPG

Re: TFTPfileExist, tloop, tinfiniteloop and tsleep process

AAA999 — Tue, 26 Nov 2019 14:13:56 GMT

Can anyone please provide a solution would be helpful.

Thanks

Re: TFTPfileExist, tloop, tinfiniteloop and tsleep process

Anonymous — Wed, 27 Nov 2019 02:00:29 GMT

quick question: how many times do you want the job to re-check the file if it is not present? You need to set a number and exits the loop after retrying several times, otherwise, the job will execute the loop all the time if the file is not present.

Re: TFTPfileExist, tloop, tinfiniteloop and tsleep process

AAA999 — Wed, 27 Nov 2019 17:29:37 GMT

Want to check the file every 1 minute for 15 minutes. If the file is not present after 15 minutes then only send the email. In my current scenario it keeps sending email every 1 minute if the file is not present. The process should stop once the file has been processed and loop again next day. Presently I am using For condition in tloop and not passing any parameters in trun condition.

loop1.JPG
ftpchild.JPG
email_condition.JPG

Re: TFTPfileExist, tloop, tinfiniteloop and tsleep process

AAA999 — Mon, 02 Dec 2019 16:28:43 GMT

Can anyone please provide a solution?

Re: TFTPfileExist, tloop, tinfiniteloop and tsleep process

Anonymous — Mon, 02 Dec 2019 22:27:18 GMT

First of all, I think a Talend Job is wrong for this requirement. Talend Jobs are batch. They start, do a job and end. If you want something to be continually polling for a file, it might be better to use Talend ESB. However, since you are close to your requirement here, I'll start with a requirement to ....

1. Poll for a file for 15 mins

2. If the file is found, some action takes place

3. If the file is not found after 15 minutes, send an email

The tLoop with the tRunJob and tSleep at the end looks like a good start. The adjustments I would make are these.....

1. The job run by the tRunJob needs to check for a file and if the file is found, do whatever task is required THEN submit a success response using a tBufferOutput (check the Talend documentation for how to use that). If the file is not found, it needs to send a fail status using the tBufferOutput.

2. After the tRunJob, you need to test the result of the tBufferOutput (which returns a data row from the tRunJob). This needs to set a variable. Maybe a globalMap variable. This variable needs to be used in your loop condition. If the file is found, the loop should end as the task is complete. Otherwise, the loop should continue for 15 minutes. Assuming the tSleep is for 1 minute, that is 15 loops.
3. Use a SubJobOk link to go from the tLoop to a tjava. From the tJava use RunIf links to decide whether to send your email. If the file was found (your variable set in step 2 can tell you this) send the email. Otherwise do nothing.

Re: TFTPfileExist, tloop, tinfiniteloop and tsleep process

AAA999 — Wed, 04 Dec 2019 15:59:24 GMT

The job run by the tRunJob needs to check for a file and if the file is found, do whatever task is required THEN submit a success response using a tBufferOutput (check the Talend documentation for how to use that). If the file is not found, it needs to send a fail status using the tBufferOutput.

I tried the tBufferOutput and went through the document but I couldn't figure out after which component to add the tBufferOutput and how to send a fail status using the tBufferOutput. Also in the tRunJob I am not passing any variable (what values need to be passed). In the job I created I am adding a "run if" condition for file not exist link from tFTPFileExist which I believe I should remove as it keep sending an email after every minute. For success the email component is in the end.

Returns a data row from the tRunJob (how to pass this). the loop should continue for 15 minutes. Assuming the tSleep is for 1 minute, that is 15 loops.?

3. Use a SubJobOk link to go from the tLoop to a tjava. From the tJava use RunIf links to decide whether to send your email. If the file was found (your variable set in step 2 can tell you this) send the email. Otherwise do nothing.

tloop --> tJava (parent job?)

I am attaching the job I created. Please advice what i am missing.

Thanks

tchild.JPG
tWaitforfile loop.JPG

Re: TFTPfileExist, tloop, tinfiniteloop and tsleep process

Anonymous — Wed, 04 Dec 2019 17:48:23 GMT

OK, I am a little stuck following your job design here. A best practice for writing jobs is to have the components ordered on your job in the same way words are ordered on a page. Top to bottom and left to right. Your subjobs (continuous coloured areas or components joined using OnComponentOk links) should be listed with the first subjob to run at the top and they should be connected via an OnSubJobOK link from first component of a subjob to first component of the next subjob. Like this....

This is just an example of the flow of components. This is just a job I have been working on.

With regard to your job you need to separate the checking for files and processing found files from the sending of the email. The sending of the email should be in the outer tLoop job. The child job should be handling the checking for the file and the processing of found files OR identifying that the file has not been found. Your child job needs to have two flows starting with checking that the files exists. If it does, then you process the file. If it does not, then you use something like a tFixedFlowInput component to send a failure message to a tBufferOutput. This way, the outer job can check the tRunJob after every loop. If the tRunJob has a failure message 15 times, then you stop the loop and send the mail.

Re: TFTPfileExist, tloop, tinfiniteloop and tsleep process

AAA999 — Tue, 10 Dec 2019 19:29:27 GMT

I have modified the design and able to pass the values from child to parent job. The process does not exit even if the file has been processed. I want the process to exit and check again for the file next day. Any suggestion. I have attached the updated design.

parentjobverify_trunjob.JPG
childjobverifyifixedflowifcondition.JPG
parentjobverifytjavarowcondition.JPG
parentjobverify.JPG
childjobverify.JPG

Re: TFTPfileExist, tloop, tinfiniteloop and tsleep process

AAA999 — Tue, 10 Dec 2019 19:30:06 GMT

Attachment

parentjobverify_trunjob.JPG
childjobverifyifixedflowifcondition.JPG
parentjobverifytjavarowcondition.JPG
parentjobverify.JPG
childjobverify.JPG

Re: TFTPfileExist, tloop, tinfiniteloop and tsleep process

Anonymous — Wed, 11 Dec 2019 10:19:30 GMT

Remind me, what version are you using? I cannot see enough from the screenshots. Export your job and attach it and I will have a look at it. That might be easier.

Re: TFTPfileExist, tloop, tinfiniteloop and tsleep process

AAA999 — Wed, 11 Dec 2019 14:56:40 GMT

Attached job

j_child.zip
j_parent.zip

Re: TFTPfileExist, tloop, tinfiniteloop and tsleep process

Anonymous — Wed, 11 Dec 2019 22:05:08 GMT

Right, I think I have it. I cannot test this here so I have taken a screenshot of the parent job. I reckon this can be sorted with just a few changes there. Here is the new Parent Job....

I've added a new tJava component, modified the tLoop expression and edited the tJavaRow code. Everything else remains the same.

tJava Code

globalMap.put("check", true);

tLoop Config

tJavaRow Code

//Code generated according to input schema and output schema
System.out.println("The File Message:"+input_row.FILENAME);

globalMap.put("check", false);

if(input_row.FILENAME){
	globalMap.put("check", true);
}

What this does is make use of the boolean you are sending back from the Child Job (in a field called FILENAME??). After every iteration, the "check" globalMap variable is set to false. This will stop the tLoop. However, if the child job returns a value that is true (you are converting a false to a true in your child job), then it sets the "check" value back to true. This will keep the tLoop going.

The Child Job does not necessarily have the logic that I would have chosen, but I have given you code that only affects the Parent Job to make it as easy as possible to fix this. You may want to (once you see how this works) refactor your Child Job to make the whole process a bit more logical with regard to the messages sent back from the child job. Maybe you could send a true or a false based on the success or failure of the find file?