How to get the last 2 days file from the remote ftp site?

[Haresh]

I am trying to retrieve the last 2 days worth of files from the remote ftp site when they have 5000 odd files present in the sftp site.I just want to sort the file list by desc order and get the last 2 days files alone into the local folder.

i am thinking if there is any way to implement it without using tbufferoutput and iterating through all the list of the files(5000) in the ftp site.Is there a quicker way to do this ,please help.

Please see the job flow below:

i am trying to read the ftpfile properties like the date and sort by desc and give the range and fetch the latest 2 days only.

Urgently need a better solution

[Haresh]

@Shicong Hong​ thanks for your response,i kind of overlooked it...This is my complete flow:

i had to integrate tjavarow to store the value of row2.abs_path in a globalmap and read them in the tftpget filemask like this :

((String)globalMap.get("abs_path"))

Because whenever i tried to directly link the tmap to tftpget iteration,i always got the value as null for abs_path.Is this the best possible approach or should i tweak anything in these to finally download the files extracted from the tMap

[Anonymous]

the global variable ​((String)globalMap.get("abs_path")) should be used in next subjob. I would suggest to filter all the files and store the file path to memory, get the files in next subjob. eg:

tFTPFileList.....tMap---tHashOutput

|onsubjobok

tHashInput--main--tFlowToIterate--iterte--tFTPGet.

[houssem_B]

@Haresh Sivakumar​  can you send us a screen shot of your tftpfilepropreties schema?

[Haresh]

interesting enough i cant seem to find thashinput/output components in my palette,although i am using talend 7.2 version.Is there any packages that i could install to fetch that component.

I can find tbufferoutput/input however

[Haresh]

i am able to fix and see the results in tmap now,the question is how do i download the filtered files using tftpget now after tmap using the flow you had mentioned.

tftpget always returns null and nullpointer exception when trying to download the files based on the tmap filter criteria....should i add anything else in between?

[Haresh]

Ok i was able to find and install thashinput/output but however i am running into issues when trying to run the job,it says the jar file is missing...any idea where i can install the thash jar files to make them compatible

[Haresh]

I tried changing my flow using tbufferin/out and i could see 30 rows are coming through,but when i try to retrieve using tftpget ,i am seeing only null values?Not sure why!

And this is the filemask i am using in tftpget :

((String)globalMap.get("row2.abs_path"))

[houssem_B]

@Haresh Sivakumar​  try this in your component propreties

Dont forget C:/

if it works I can show you the next steps

-----------------------------------------------------------

ont forget to like the solution if you find it helpful

Thanks 😁 😁

[houssem_B]

@Haresh Sivakumar​ so after filtring your files you can do as below

-----------------------------------------------------------

ont forget to like the solution if you find it helpful

Thanks 😁 😁

[Haresh]

are you sure it has to be basename,because i am getting the value as filename.csv whereas only if i use abs_path parameter it retrieves the whole file path like users/desktop/test/filename.csv and btw i can only see today's file being retrieved by this method,not yesterday or day before!

Thanks for your inputs