topic Re: Move (cut) files instead copy ftp to local folder in Talend Studio

Move (cut) files instead copy ftp to local folder

JavierP2 — Sat, 16 Nov 2024 02:37:34 GMT

Hi there,

I'm working on a program that moves files between a SFTP server and a local folder.

It works fine but I have a problem, seems as the tFTPGet copy the files from the SFTP and move them to the local folder but the files still remain into the SFTP. I do need to cut the files instead copy, is there any way?

I've tried with a tFTPDelete but this solution doens't work 100% as is for example during the process between the FTPGet and the tFTPDelete a new file comes it auto deletes as is on the SFTP and it's not copied.

So solutions that I think that can work:

1) Cut the files instead copy

2) Any rule into the tFTPDelete to only delete the ones listed into the tFTPFileList

Any ideas?

Thanks a lot!

Re: Move (cut) files instead copy ftp to local folder

billimmer — Tue, 21 Apr 2020 20:15:55 GMT

This works for me:

tFTPFileList-----(iterate)---->tFTPGet----(componentOK)---->tFTPDelete

Delete the files as you get them.

Re: Move (cut) files instead copy ftp to local folder

JavierP2 — Tue, 21 Apr 2020 20:26:23 GMT

thanks for the reply.

the thing is that I have other works before the delete so in that time that the work runs the system can generate other file (i've tested it) so in any way I need to ensure that it only delete the ones already copied.

Re: Move (cut) files instead copy ftp to local folder

billimmer — Tue, 21 Apr 2020 20:32:08 GMT

Why not download the files and then do your 'other work' on the local copies?

If the 'other work' has to be on the ftp server, then rename the files, and do the other work on the renamed copies.

Re: Move (cut) files instead copy ftp to local folder

JavierP2 — Tue, 21 Apr 2020 20:44:53 GMT

Rename the names make sense, I've tried before with tFTPRename but I don't know how to rename the files keeping the same name but adding some extra at the begining, like a OK_+"FILENAME" any idea how to do that?

If I put "" into the newname (on the tFTPRename) it keeps the same but cannot add the OK_ at the begininnig.

Thanks !

Re: Move (cut) files instead copy ftp to local folder

billimmer — Tue, 21 Apr 2020 21:08:39 GMT

don't you have the name from your tFTPFileList for use in your tFTPGet? Maybe

"OK_" + ((String)globalMap.get("tFTPFileList_1_CURRENT_FILE"))

Re: Move (cut) files instead copy ftp to local folder

JavierP2 — Tue, 21 Apr 2020 22:17:24 GMT

Thanks for the reply.

Tried to put it after the FTPGet but it shows me this error:

Exception in component tFTPRename_1 (MOVEORDERSDV)
3: Permission denied

Do I have to declare something into the tFTPlist?

Re: Move (cut) files instead copy ftp to local folder

billimmer — Tue, 21 Apr 2020 22:26:12 GMT

try in the filemask:

((String)globalMap.get("tFTPFileList_1_CURRENT_FILE"))

instead of

"*"

To rename just that file.

Also, I would rename it first, and then "get" the copy.

Re: Move (cut) files instead copy ftp to local folder

billimmer — Tue, 21 Apr 2020 22:33:39 GMT

Also, I don't understand why you are using a OnSubjobOK with your tFTPFileList, and then have 4 tFTPGet's. That implies that you "know" the file names and don't need the tFTPFileList to get the file names.

I mention this because the ((String)globalMap.get("tFTPFileList_1_CURRENT_FILE")) is only going to be the last file if finds, because it's used after the OnSubjobOK.

sorry, I just don't understand what you are trying to do.

Re: Move (cut) files instead copy ftp to local folder

JavierP2 — Tue, 21 Apr 2020 22:34:55 GMT

It works but only with the first file, If I move more than 1 it only renames the first one.

Re: Move (cut) files instead copy ftp to local folder

billimmer — Tue, 21 Apr 2020 22:37:07 GMT

then shouldn't there be an iterate connecting the tFTPFileList to a tFTPGet. And let it iterate over your files so you don't need 4 gets?

Re: Move (cut) files instead copy ftp to local folder

JavierP2 — Tue, 21 Apr 2020 22:53:16 GMT

I will try to explain the full procedure, so maybe it can makes sense to you and maybe there's an easier way to do it (it's my 3 time with talend).

So I have SFTP Folder where I will receive many files during the day.

With this files I want to run every "x" mins this program that have to do the following:

1) Copy the files from the SFTP that start with "XX" into 1 local folder

2) Copy the files from the SFTP that start's with "BB" into other local folder

3) Once copied to the 2 folders I want to rename these files from the SFTP (with "OK_+FILENAME") and move them to 2 different "Historic" local folders. (that's why there's 4 gets) cause I have 4 different names and folders to move.

4) Remove the files from the SFTP.

5) Send a confirmation mail (this works).

Re: Move (cut) files instead copy ftp to local folder

billimmer — Tue, 21 Apr 2020 23:16:45 GMT

That all makes sense.

First rename everything "OK_" + so you can keep track of what you are doing

tFTPFileList_1----(iterate)---->tFTPRename---(OnComponentOK)----tFTPDelete

Now you are only working on "OK_" files and don't need to track new stuff. Next build your email list for only files that start with "OK_", just like you are already doing, but set the tFTPFileList_2 to "OK_*"

tFTPFileList_2----(iterate)---->tIterateToFlow---(main)----tLogRow---(main)--->tJavaFlex

Now you are ready to do work Iterate through the "OK_XX*" files and do your work. Use a filemask in the tFTPFileList so you only get these files. copy them to local, archive to history folder, and delete each one.

tFTPFileList_3----(iterate)---->tFTPGet---(OnComponentOK)---->tFTPRename---(OnComponentOK)---->tFTPDelete

Now do the same for the "OK_BB" files

Finally, send your email

Re: Move (cut) files instead copy ftp to local folder

JavierP2 — Tue, 21 Apr 2020 23:24:06 GMT

Thanks for the reply, youre helping me a lot.

I need to change 1 thing, for the first movement, the files cannot have the OK_ added, So OK_ it's only for the files that will go to the historical folder.

It makes any change on your idea? I will try to replicate.

Thanks again!

Re: Move (cut) files instead copy ftp to local folder

billimmer — Wed, 22 Apr 2020 14:22:44 GMT

You can strip off the "OK_" after you copy it to local.

Or instead of adding "OK_", you can move the files to a "OK" folder, and then do all your work from there.

Re: Move (cut) files instead copy ftp to local folder

JavierP2 — Wed, 22 Apr 2020 19:45:38 GMT

Hi,

I've managed to do it with the renaming at the begining, I think this way there's less time between jobs so it would be more 'safe'.

I have a last question, it seems as work but in any way I need to close the connection with the FTP and also send the email.

So I want in any way to connect the tFTPDelete_2 to the tFTPClose when there's no more files to process and also send the email. Not sure how to connect.

I've thought about an if into the tFTPDelete Like : ((Integer)globalMap.get("tFTPFileList_4_NB_FILE"))==0

But it's not working.

Any ideas how to connect that?

Re: Move (cut) files instead copy ftp to local folder

billimmer — Wed, 22 Apr 2020 19:50:58 GMT

you should connect each subjob. So connect

tFTPFileList_1 to tFTPFileList_2 with a OnSubjobOK

tFTPFileList_2 to tFTPFileList_3 with a OnSubjobOK

tFTPFileList_3 to tFTPFileList_4 with a OnSubjobOK

tFTPFileList_4 to tFTPClose_1 with a OnSubjobOK

Re: Move (cut) files instead copy ftp to local folder

JavierP2 — Wed, 22 Apr 2020 20:23:07 GMT

Worked like a charm.

Thanks a lot, you saved my day!!