Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hello
I would like to set up a process for recovering ftp files with the certainty that on the remote ftp there are all the sub files of each meta file.
a meta file is an xml file consisting of a list of data file names
example:
d20200505_metafile_0001.xml
<root>
-<responses>
--<filename>d20200505_filedata_0001.xml</filename>
--<filename>d20200505_filedata_0002.xml</filename>
--<filename>d20200505_filedata_0003.xml</filename>
-</responses>
d20200505_metafile_0002.xml
<root>
-<responses>
--<filename>d20200505_filedata_0004.xml</filename>
--<filename>d20200505_filedata_0005.xml</filename>
-</responses>
I have already managed the ftp connection part, the list of meta files (tFTPFileList), recover a local copy (tFTPGet) meta files, list local meta files (tFileList).
then I wanted to extract each xml element "/ root / responses / filename" (tFileInputXML) by meta file, and do a
(tFTPFileExist) on each file name mentioned in the meta file, and if all the files are present in this meta file we download them. otherwise if one xml data file is not present then we stop this process and we delete the meta file which refers to it locally. and do this for each meta file.
I get stuck on how to isolate the process from each meta file. Is it possible to do it with Talend components without having to do tJava and do everything in java.
thank you
@martel
If you use an existing FTP connection on tFTPFileExist, make sure the connection is created before it is used, so change the job to:
tFTPConnection
|onsubjobok
other subjobs.
Hi
As you do now, download all meta files to local system, and then iterate each meta file, pass the current file path to a child job, refer to this article to know how to pass a value from main job to child job.
https://community.talend.com/t5/Design-and-Development/Passing-a-value-from-a-parent-Job-to-a-child-...
In child job, create a job as below.
tFileInputXML1: Read all files from current meta file and store them to memory for used later.
on tJava2, check the file if it exists on FTP or not, if it does not exist, break the loop and set the context variable context.needDownload (which is defined in child job and set it as true by default) to false;
if(((Boolean)globalMap.get("tFTPFileExist_1_EXISTS"))==false){
context.needDownload=false;
break;
}
set the condition of runIf1 as:
context.needDownload
set the condition of runIf2 as:
!context.needDownload
tFileDelete: delete the current meta file if one of sub file does not exist on FTP.
Regards
Shong
mhh ok,
1) it not possible to re-use the ftpConnect into the job child ?
if i have a tFTPConnect_1 for the job parent, connect ok, download the meta files ok.
but into job child (i cant use the connection ftp ?)
java.net.ConnectionException : Connection refused : connect.
how can say in job child continue to use the connection ?
im obliged to close FTPConnect in my job parent before call the tRunJob ?
thanks you
when i run job parent, i dont see the stats in windows job child ?
About the NPE on tFTPFileExist, change the expression on FileName Field to
(String)globalMap.get("row2.compressFileName")
Hey,
i still have problem with tFTPFileExist with nullPointerException.
1) tHasInput read the first line with the scema only compressedFileName i see in debug mode (cf screen join)
-> tFlowToIterate (key-value save on global cheked);
-> tFTPFileExist with ((String)globalMap.get("row2.compressedFileName")) in file name field.
thanks you.
I realize that it is not so simple to do something on talend. But surely a question of habit (we have the complete TAC platform, I dare not imagine the rest at the learning level.)
For me your tFTPFileExist return allways null
-
----------------------------------------
---------------------------------------------------------------------------
My file "at.......................Response.0000000001.xml.gz" exist , but the component return null ?
i use 7.2.1 Talend
ok, this little example works.
but your first screen to explain the flow with iterating call tFTPFilesExist return always nullpointerexception ?
I have the list of filename (here 3), but when i read the first record in hashInput -> tFlowToIterate_1 -> tFTPFileExist this throw an exception nullpoint.