[resolved] Read files from ftp site and load to postgresql db

[Anonymous]

Hi ,
I have a small requirement to perform. Can any one help me with the steps to perform this.I dont know how to log the list of files in to a log table and update it based on loading status.
       I need to Create a  ETL process to generate fact data by resolving the lookup.I have to read the files from a ftp site and load the data to a local Postgresql database with following features.
      Before loading the data into database, log the list of files in log table and update its status (Downloaded , Processed , Failed)
·  Any duplicate file should be notified via email.(no duplicate files to be loaded)
·  Process can run only if all files are available.
·  Notification of success and failure of job along with Job log.
·  Propose a quality report for daily data load.
Expecting a help.
Thanks and Regards,
Naksh.

[Anonymous]

Hi Sabrina,
Please find the job settings.
Thanks and Regards,
Naksh

[Anonymous]

Hi,

Yes i had provided the filepath variable .The issue which i was facing is instead of iterate, by mistake i have chose row connection as component ok. Thats why the null pointer error was showing.When i changed it to iterate,null pointer issue got resolved.
But now I am facing another issue. The job had run succesfully without any errors. It iterated 12 files from remote server and job ended without any errors. But the files did not get copied to my local directory. The destination is still empty.

When you put global variable((String)globalMap.get("tFTPFileList_1_CURRENT_FILEPATH")) in file mask entry of tFtpGet, no file is saved in your local?
Have you checked out "Die on error" option in tFTPGet to see if there is any error info printed on console?
Best regards
Sabrina

[Anonymous]

Hi Sabrina,
I have checked out Die on error option also.No file is saving in my local. Instead throwing 
' Exception in component tFTPGet_1
com.enterprisedt.net.ftp.FTPException: 550 Failed to change directory'
Thanks and Regards,
Naksh

[Anonymous]

Hi,
For testing, could you please try to deactivate all of components excepts tFTPGet(for log component) and create the FTP connection on the component itself, instead of using an existing connection, to see if it works.
Best regards
Sabrina

[Anonymous]

Hi Sabrina,
I deactivated all components except tFTPGet and ran the job. Still its throwing error:
Exception in component tFTPGet_1
com.enterprisedt.net.ftp.FTPException: 550 Failed to change directory.
Thanks and Regards,
Naksh

[Anonymous]

Hi,
Could you please show us tFTPGet component setting screenshot?
Best regards
Sabrina

[Anonymous]

Hi Sabrina,
Please find the tFTPGet settings

[Anonymous]

Hi,
Could yo please use "/incoming/" instead of ((String)globalMap.get("tFTPFileList_1_CURRENT_FILEPATH")) in Remote directory? We have deactivated component tFTPFileList in your job design. There is only one component tFtpGet in your work flow.
Best regards
Sabrina

[Anonymous · Accepted solution]

Hi Sabrina,
Oh yes!! Careless mistake by me. Thanks for the reply. Now its downloading files to my local.
I activated all other components and in tFTPGet, i left the remote directory as "/incoming/" only
Now my job works fine.
Thanks Sabrina. U rocks!!
Regards,
Naksh

[Anonymous]

Hi
That's really great to know that! Could you please mark this topic as resolved so that other community users will be informed that this thread has been resolved?
Best regards
Sabrina