topic Re: Retrieveing files from FTP (javacode in use) in Talend Studio

Retrieveing files from FTP (javacode in use)

Anonymous — Sat, 16 Nov 2024 13:04:38 GMT

Hi
I have to retrieve the most recent version of my files from my FTP which are comming daily.
Every day our data source getting different files folder with a datestamp appended as the filename most in format of date in format "yyyyMMdd", when are sent out .
e.g:
On the top level I am having file name and under that name having a few different files in format of date see below.
20110201
20110202
20110203

I'm assuming I'll need to use a tFTPconnection, tFTPFileList and potentially tFileProperties to look for all files . However, when I get my files from FTP I will need to save that files to directory which should be somewhere under my 😧 or what ever drive.
Is anyone can help me with this.
Thanks
Zeljka

Re: Retrieveing files from FTP (javacode in use)

Anonymous — Wed, 23 Feb 2011 11:51:08 GMT

use tFTPGet to download the files from FTP and save in the local folder.

Re: Retrieveing files from FTP (javacode in use)

Anonymous — Wed, 23 Feb 2011 13:12:55 GMT

hi,
Thanks for this,
yes I have used tFTPGet to get my folder from FTP and after tFTPGet I am having tFTPFilelist which I thought I can use to read files under.
however as I am having a few folders under I try to filter but not working.
Not sure what I can use between my tFTPGet and tFTPfilelist to get my files as I am having this order folders:
folderFilles ----- folders with names like 20110201 ......20110210

and below is image of my work...

Re: Retrieveing files from FTP (javacode in use)

Anonymous — Wed, 23 Feb 2011 17:11:27 GMT

In my views there is a problem with your job.
You need to use TFTPFileList--Iterate-->tFTPFileProperties--tJavaRow-----if-->tFTPGet
get the file properties and do your validation. if Correct assign the result to a context variable(boolean).Check the context in the if condition and then download the file to local folder.
Use the below for getting the current file.
((String)globalMap.get("tFTPFileList_1_CURRENT_FILE"))
Hope this clarifies.

Re: Retrieveing files from FTP (javacode in use)

Anonymous — Thu, 24 Feb 2011 09:36:56 GMT

Hi ,
Thank you,
I have change my job in the order as you suggested and giving me all my files as should .
However, I am having every day files with probably same name but the only differents are the time stamp and I am not sure how I can get my files with latest time stamp.
I would like to have the only files which are having the new time when are send out as I need to retrieved the only that files even if having same name.
Does anyone have idea how I can do this .

Zeljka

Re: Retrieveing files from FTP (javacode in use)

Anonymous — Thu, 24 Feb 2011 10:50:46 GMT

Each time you download a file, save the file name in a CSV or database.
In the given scenario, pass the tJavaRow-->tMap--tJavaRow---if(true)--tFTPGet
have a lookup of the fileName in the CSV or database. If it doesnot matches download the file.
There are other ways too like once a file is downloaded u put the file in sepearte folder in FTP, or dont keep the files in FTP after download, As you can keep the download files in your backup folder in local.
Hope it helps you

Re: Retrieveing files from FTP (javacode in use)

Anonymous — Thu, 24 Feb 2011 12:16:14 GMT

could you please give me some screenshots
Thanks
Zeljka

Re: Retrieveing files from FTP (javacode in use)

Anonymous — Thu, 24 Feb 2011 16:03:46 GMT

Hi
I have uploaded some screenshots which will explain.
In the tMap you should enable inner join.
After the tJavaRow, put an if condition as if there is no file then no need to run tFTPPut.
Once a file is downloaded you can append the CSV file and add the new name to it.
Hope this helps you

Re: Retrieveing files from FTP (javacode in use)

Anonymous — Thu, 24 Feb 2011 16:52:13 GMT

Thank you ,
Zeljka

Re: Retrieveing files from FTP (javacode in use)

Anonymous — Fri, 25 Feb 2011 09:48:46 GMT

Hi,
I am having problem to figure out how to pull out the files by looking only on the time stamp as I need to have files with latest time stamp or also case were I have to look for files which are older like from last month etc..
I can't use name of the file as in most of cases I am having files with same name , the only different between all these files are as I keep saying the time stamp where are sent out.
Also I will need to find option how to find files like between two different time.. e.g. file with time stamp between this time stamp and that time stamp.
I was able to create the simple schema which giving me list of the file with "yesterday" time stamp or day before but not sure how to do the other option.
I hope anyone can help me.
Below is my simple schema with java code which giving me files from day before .