topic Re: How to connect multiple FTP locations in the same job in Talend Studio

How to connect multiple FTP locations in the same job

Anonymous — Fri, 12 Oct 2018 12:41:16 GMT

I have one table which contains many distributors and retailers ftp connection details.

The table structure is as follows:

Dist_Id Reta_Id Ftp_host Ftp_user Ftp_pswd
------------------------------------------------------------------------------------
1 a 123 abc xyz
1 b 456 def mno
2 c 431 ghi p9r
3 d 789 jkl 12e

So for getting all files for Dist_Id=1, I want to connect ftp locations of Retailers a and b. And download files in each ftp location to the same folder.
How can I connect to multiple ftp locations and download files in the same folder??

Re: How to connect multiple FTP locations in the same job

fdenis — Fri, 12 Oct 2018 14:41:55 GMT

create your ftp job for one connection one file (use context to store values of line 1)
create a job calling this job (trunjob) with context values from your table.

Re: How to connect multiple FTP locations in the same job

Anonymous — Sat, 13 Oct 2018 05:20:50 GMT

Thanks for your response.. There is a chance for coming new retailers for each distributor.. At that case, want to connect to that retailer's ftp location automatically and download the files.. Could you please make it more clear with some examples...

Re: How to connect multiple FTP locations in the same job

Anonymous — Sat, 13 Oct 2018 06:35:41 GMT

Neena,

below is the example which Denis was trying to explain.

consider that the ftp details are there in a file. so use tfileinputDelimited and connect to tFlowToIterate which would iterate the tRunJob for every output row from the file.

The advantage of tFlowToIterate is that it sets the inflow values in global variable as key-value pair. In tRunJob, pass all the parameters from tFlowToIterate to context variable of the child job.

In the child job you can have FTP component. use all the defined context variables in the ftp component.

Hope this helps!

Re: How to connect multiple FTP locations in the same job

Anonymous — Mon, 15 Oct 2018 08:42:27 GMT

Thanks for your reply.. It is very much helpful.. I Designed the job based on your reply.. But when connects to second ftp location it shows some exceptions :

Exception in component tRunJob_1
java.lang.RuntimeException: Child job running failed.
com.enterprisedt.net.ftp.ControlChannelIOException: Control channel unexpectedly closed ('' read so far)

Is there any solution??

Re: How to connect multiple FTP locations in the same job

fdenis — Mon, 15 Oct 2018 09:20:34 GMT

try the second job with second ftp context.

Re: How to connect multiple FTP locations in the same job

Anonymous — Mon, 15 Oct 2018 10:29:22 GMT

I think the second one is a SFTP protocol. So how to handle both FTP and SFTP in the same job??

Re: How to connect multiple FTP locations in the same job

fdenis — Mon, 15 Oct 2018 14:56:37 GMT

add a sftp subjob and switch defore or add sftp connection into subjob and switch inside subjob

Re: How to connect multiple FTP locations in the same job

Anonymous — Mon, 15 Oct 2018 16:52:55 GMT

Neena,

You can do this many ways. Here is the easiest way I would recommend. First of all, you should send the FTP method (either FTP or SFTP) from the parent to tRunJob.

Here is the screenshot of child job

tJava: just to print the FTP method

First order: connect tJava with runif to FTP component.

Second order: connect tJava with runif

tFtp get with "SFTP support" enabled. depending on your design , you can choose the "Authentication method" to public key. provide key Passphrase and private key (if required).

The idea is to have runif and decide your flow based on the ftp method sent by your parent.

Hope this helps!

Re: How to connect multiple FTP locations in the same job

Anonymous — Tue, 16 Oct 2018 06:46:41 GMT

Very Thanks for your reply... It works ....

Re: How to connect multiple FTP locations in the same job

Anonymous — Tue, 16 Oct 2018 08:49:24 GMT

Hi,

I have one more block to complete this job..

While using context variable context.FTP_Files as Filemask in FtpGet component, it did not download files when more than one file name are provided in the FTP_Files column separated by comma.But downloads when only one file name is provided. How can I download all the files provided in the FTP_Files column??

Example:

Dist_Id Reta_Id FTP_Files

---------------------------------------------------------------

1 a ab.csv,mn.csv

1 b pq.csv

In this case, for Dist_Id=1, context.FTP_Files only download pq.csv. Actually I want to download all these 3 csv files in the same folder.. (Dist_Id is same).

Thanks In Advance..

Re: How to connect multiple FTP locations in the same job

fdenis — Tue, 16 Oct 2018 10:22:47 GMT

filters are list you cannot dynamically convert strings to list.
so you have to use a tNormalise to split your file liste.

Re: How to connect multiple FTP locations in the same job

Anonymous — Tue, 16 Oct 2018 18:58:04 GMT

Neena,

below is the job with tNormalize as Denis suggested.

row 1 - settings:

tNormalize - component settings

Output

Input

Regards,

Ragu

Re: How to connect multiple FTP locations in the same job

Anonymous — Wed, 17 Oct 2018 06:39:02 GMT

Thanks for both of you.. It Works...

Re: How to connect multiple FTP locations in the same job

Anonymous — Wed, 17 Oct 2018 07:14:07 GMT

Hi,

I have one more doubt..

When I download all retailer files and inserted to one common table, it can not distinguish which data is getting from which Retailer.. So I want to append one column(Reta_Id) in all csv files when the file is downloaded from ftp location. Is it possible to append a column in csv at the time of downloading from FTP??

Thanks In advance

Re: How to connect multiple FTP locations in the same job

fdenis — Wed, 17 Oct 2018 09:46:31 GMT

yes you can but you have to download then open and add column.

Re: How to connect multiple FTP locations in the same job

Anonymous — Wed, 17 Oct 2018 10:35:46 GMT

Thanks for your reply...

I want to add the Retailer_Id column in each file when it is downloaded.. Then only corresponding Retailer_Ids can be added. Since there are many files Manually open and adding the column is not possible..