topic Downloading only certain directories from an FTP server in Talend Studio

Downloading only certain directories from an FTP server

Anonymous — Mon, 30 Jul 2018 14:15:33 GMT

Hello Talend Users,

I have a FTP directories with many subdirectories (representing years from 1999 to 2018) . For now I can only download all files in all folders. However I want to only download the folders from 2011 to 2018.

So far my Job looks like this:

How can I iterate over the Folders from 2011 to 2018? With tForeach? And if yes how can I connect it to FTPGet?

Thanks in Advance.

Re: Downloading only certain directories from an FTP server

Anonymous — Tue, 31 Jul 2018 05:24:46 GMT

Hello,

Do you want to list sub-directories in FTP folder?

Could you please let us know if this resolved topichttps://community.talend.com/t5/Design-and-Development/Exclude-File-Mask-in-TFTPFileList/m-p/124806 is OK with you?

Best regards

Sabrina

Re: Downloading only certain directories from an FTP server

Anonymous — Tue, 31 Jul 2018 08:19:22 GMT

This is not really the solution to my problem as it is about excluding certain files from the download. I however want to download only the directories 2011 to 2018 in a list of directories which reaches from 1999 to 2018. So somehow I have to provide a list to the FTPget job.

Best regards
Maximilian

The FTP Directory looks like this:

Re: Downloading only certain directories from an FTP server

iamabhishek — Tue, 31 Jul 2018 12:41:53 GMT

You could use tSSH to filter out the folders using unix commands. The output of tSSH you could then store in global variable which is tSSH_1_STDOUT

Once you have the list of folders in the global variable just use them to iterate over using

tFTPFileList -> iterate ->tFTPGet

Sample unix command to filter out folders starting with "talend" -

"find $(pwd) -maxdepth 1 -type d -name \"talend*\""

Re: Downloading only certain directories from an FTP server

Anonymous — Tue, 31 Jul 2018 13:52:36 GMT

Hi,

ok sounds really complicated to me since I am a Talend Beginner. How do I connect the tSSH job to the FTPfilelist?

Re: Downloading only certain directories from an FTP server

iamabhishek — Tue, 31 Jul 2018 14:34:07 GMT

tSSH - > on subjob ok -> tFTPFileList
It might sound complicate but it's not. The solution is not straight forward because FTP support is with files and not with folders - that's the reason for the tweak.