topic Re: Problem with FTPPut when deleting the files afterwards in Talend Studio

Problem with FTPPut when deleting the files afterwards

Anonymous — Tue, 28 Jan 2014 10:57:23 GMT

Hi everyone,
I'm getting a strange behaviour, and I think I'm doing something wrong in the configuration, so any help would be appreciated.
Here is the scenario:
I'm getting two files from a FTP.
I transform them and put them back on the same FTP.
Afterwards, I'm deleting the temporary files on my server.
So the job looks like this (simplified version):

// Get the files
tFTPGet -> 
// Transform and put first file (lets call it subjob1)
tFileList_1 -> tFileInputDelimited -> tMap -> tFTPPut --onComponentOk->
// Transform and put second file (subjob2)
tFileList_2 -> tFileInputDelimited -> tMap -> tFTPPut --onSubjobOk->
// Delete temporary files (subjob3)
tFileList_3 -> tFileDelete

When i'm running this job, I get the first file uploaded correctly (from subjob1), but not the second one (from subjob2).
If I disable the subjob3 (tFileList_3 -> tFileDelete), I get the two files in my FTP folder...
So my question is, why is the subjob3 triggered when the subjob2 is not finished?
What am I doing wrong here?
The solution I found (really ugly) is to add a "tFTPPut" in the subjob2, which generates an error "No files match", but at least the files are uploaded to my FTP directory:

// Transform and put second file with hacking
tFileList_2 -> tFileInputDelimited -> tMap -> tFTPPut -> tFTPPut_2 --onSubjobOk->

This works, but I would like to find what's the correct solution...
I'm using Talend Open Studio for Data Integration (5.1.1.r84309).
Thanks for you help.

Note: I also tried changing onSubjobOk by onComponentOk between subjob2 and subjob3, same results, the files is not uploaded without the hacking.

Re: Problem with FTPPut when deleting the files afterwards

Anonymous — Tue, 28 Jan 2014 16:13:15 GMT

I am not certain but you may try only using onSubjobOK and not using onComponentOK at all.
I suspect there is an issue with subjob 2 and 3 running based on a component of subjob 1.

Re: Problem with FTPPut when deleting the files afterwards

Anonymous — Tue, 28 Jan 2014 17:28:26 GMT

Mike,
The problem is (i described the job wrongly sorry) that I don't have 'onSubjobOk' after 'tFileOutputDelimited', i only have 'onComponentOk'..
Both subjobs looks like this:
tFileList_2 -> tFileInputDelimited -> tMap -> tFileOutputDelimited -onComponentOk-> tFTPPut --onSubjobOk->
So I can't apply your solution
Should I put something in between tFileOutputDelimited and tFTPPut?
If yes, which component?
Thank you for your help.

Re: Problem with FTPPut when deleting the files afterwards

Anonymous — Tue, 28 Jan 2014 22:46:58 GMT

Hi Mat77,
A subjob is everything whats uderlayed with the blue rectangle. A subjob is a complete Java method and proceed a flow or sometimes also an iteration.
The trigger OnSubjobOK fires if no error happened within this subjob (with all its input and output components).
On my experience it is a better design to capsulate the processing of one file in a job (call it *_worker) and run (tRunJob) this job for every file you have (iterating through a list of files).
I would add a context variable (currentFile) of type File (File is actually a String but provides you if you set it as prompt a file chooser dialog). In the tFileInputDelimited you use this context.currentFile als filename.
Probably you have to do something similar to the output file name.
Your worker looks like:
tFileInputDelimited -> tMap -> tFileOutputDelimited
and the trigger job looks like:
tFileList_2 -- iterate --> tRunJob (worker) and set the context variable currentFile with ((String) globalMap.get("tFileList_2_CURRENT_FILEPATH"))

Re: Problem with FTPPut when deleting the files afterwards

Anonymous — Tue, 04 Feb 2014 15:55:29 GMT

Hi jlolling,
I'm sorry I didn't come back earlier to thank you for your answer, which was precise and helpful.
That's what I thought, the design is wrong, I need to do some refactoring
Thanks again for your help.