get ftp files meta and childs files (isolate process for each meta file)

[martel]

Hello
I would like to set up a process for recovering ftp files with the certainty that on the remote ftp there are all the sub files of each meta file.

a meta file is an xml file consisting of a list of data file names
example:
d20200505_metafile_0001.xml

<root>
-<responses>
--<filename>d20200505_filedata_0001.xml</filename>
--<filename>d20200505_filedata_0002.xml</filename>
--<filename>d20200505_filedata_0003.xml</filename>
-</responses>

d20200505_metafile_0002.xml

<root>
-<responses>
--<filename>d20200505_filedata_0004.xml</filename>
--<filename>d20200505_filedata_0005.xml</filename>
-</responses>

I have already managed the ftp connection part, the list of meta files (tFTPFileList), recover a local copy (tFTPGet) meta files, list local meta files (tFileList).

then I wanted to extract each xml element "/ root / responses / filename" (tFileInputXML) by meta file, and do a
(tFTPFileExist) on each file name mentioned in the meta file, and if all the files are present in this meta file we download them. otherwise if one xml data file is not present then we stop this process and we delete the meta file which refers to it locally. and do this for each meta file.

I get stuck on how to isolate the process from each meta file. Is it possible to do it with Talend components without having to do tJava and do everything in java.

thank you

[Anonymous]

Hi
As you do now, download all meta files to local system, and then iterate each meta file, pass the current file path to a child job, refer to this article to know how to pass a value from main job to child job.
https://community.talend.com/t5/Design-and-Development/Passing-a-value-from-a-parent-Job-to-a-child-...
In child job, create a job as below.

tFileInputXML1: Read all files from current meta file and store them to memory for used later.

on tJava2, check the file if it exists on FTP or not, if it does not exist, break the loop and set the context variable context.needDownload (which is defined in child job and set it as true by default) to false;

if(((Boolean)globalMap.get("tFTPFileExist_1_EXISTS"))==false){
context.needDownload=false;
break;
}

set the condition of runIf1 as:

context.needDownload

set the condition of runIf2 as:

!context.needDownload

tFileDelete: delete the current meta file if one of  sub file does not exist on FTP.

Regards

Shong

[martel]

mhh ok,

1) it not possible to re-use the ftpConnect into the job child ?

if i have a tFTPConnect_1 for the job parent, connect ok, download the meta files ok.

but into job child (i cant use the connection ftp ?) 

java.net.ConnectionException : Connection refused : connect.

how can say in job child continue to use the connection ?

im obliged to close FTPConnect in my job parent before call the tRunJob ?

thanks you

[martel]

when i run job parent, i dont see the stats in windows job child ?

2020-06-07_16-24-26.jpg
2020-06-07_16-26-36.jpg
2020-06-07_16-28-07.jpg

[Anonymous]

Hi
Yes, you need to create a new FTP connection in child job. After you download all meta files in parent job, you should close the FTP connection using tFTPClose.

Regards
Shong

[Anonymous]

About the NPE on tFTPFileExist, change the expression on FileName Field to
(String)globalMap.get("row2.compressFileName")

[martel]

Hey,

i still have problem with tFTPFileExist with nullPointerException.
1) tHasInput read the first line with the scema only compressedFileName i see in debug mode (cf screen join)

-> tFlowToIterate (key-value save on global cheked);

-> tFTPFileExist with ((String)globalMap.get("row2.compressedFileName"))   in file name field.

thanks you. 

I realize that it is not so simple to do something on talend. But surely a question of habit (we have the complete TAC platform, I dare not imagine the rest at the learning level.)

2020-06-08_08-23-38.jpg
2020-06-08_08-25-28.jpg

[martel]

 For me your tFTPFileExist return allways null

-

----------------------------------------

---------------------------------------------------------------------------

My file "at.......................Response.0000000001.xml.gz"  exist , but the component return null ?

i use 7.2.1 Talend 

[Anonymous]

Hi
The global variable should be used in subjob, so change your job to :
tFTPFileExist
|onsubjobok
tJava

on tJava, print the value of global variable, you will see the values.

[martel]

ok, this little example works.

but your first screen to explain the flow with iterating call tFTPFilesExist return always nullpointerexception ?

I have the list of filename (here 3), but when i read the first record in hashInput -> tFlowToIterate_1 -> tFTPFileExist  this throw an exception nullpoint.