How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

[Anonymous]

I was using the tFTPGet component to save folders/files from a remote server to my computer and then using tFTPPut component to transfer them to another remote location.

Occasionally, there are temporary files (~$excel file name    in my case) saved to my computer from the source. When I tried to tFTPPut the files to the remote server, the job failed. But as soon as I manually deleted the temporary file from my computer, the job succeeded.

BTW, the temporary file name actually meets the filemask requirement on the tFTPPut, but it just failed the job.

Anyone has experienced the same issue? I think I have to exclude these bad temp files either during tFTPGet or during tFTPPut. I would appreciate your help very much!

P.S.  With tFTPGet I saved folders/subfolders/files to local computer, and I used tFTPCreateDirectory to create the directories on the destination server and then used tFTPPut to transfer the files.

[Anonymous]

Is there a pattern in the name of these files, so that you can put that in the Exclude pattern in the advanced setting while you are moving them to the different server.


Best Regards,
Abhishek

[Anonymous]

Hi abhishek,

Thanks for your reply!

I am new to Talend. As I know so far, only tFlileList component has the Exclude Filemask option in Advanced setting.

Here is what I tried with tFilelist:

1. Without the Exclude Filemask, in the local directory, 9 files will be listed according to my Filemask ("*.xlsm")

    

2. When I added the Exclude Filemask ("~$*), 8 files will be listed:

3. And then I activated the tFTPPut component and ran the job, it failed:

Here is the error message -

Exception in component tFTPPut_3 (movefiles)
4:
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846)
at com.jcraft.jsch.ChannelSftp._put(ChannelSftp.java:594)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:475)
at pl_project.movefiles_1_0.movefiles.tFileList_3Process(movefiles.java:2092)
at pl_project.movefiles_1_0.movefiles.tFTPConnection_10Process(movefiles.java:2533)
at pl_project.movefiles_1_0.movefiles$4.run(movefiles.java:3796)

4. But as soon as I deleted the ~$xxxxx.slxm file from the local machine and ran the job again, it succeeded:

Thanks,

TigerMommy

with the Exclude Filemask.PNG

[TRF · Accepted solution]

Try this pattern:
tFileList-->tJava--(runIf)-->tFTPPut
Let the tJava as it comes and enter the following expression for the runIf trigger :
!((String)globalMap.get("tFileList_1_CURRENT_FILE")).startsWith("~")
tFTPPut will be called only for filenames not starting with ~

[Anonymous]

Hi TRF,

Thanks for the suggestion.

I never used tJava before. So I did as you suggested and the job failed at the tJava component -

In the tJava, I only copied and pasted your code, didn't touch other places.

And here is the error message I got -

What could be wrong?

Thanks!

[Anonymous]

Hi,

     The condition has to be given in the link (the line between two components).

      When you click on the link, in the component tab, it will show the name as If1.

 Could you please try it and let us know the result?

Warm Regards,

Nikhil Thampi

[Anonymous]

My bad -- TRF did ask me to enter the expression for the RunIf trigger but I overlooked.

Now I think the all the components were set correctly -

In my tFTPPut component, I have a filemask:

If, in my source folder, I have bad ~$ files like below among the good files, my job would still fail. But as soon as deleted the ~$All Supermarkets-TOTAL.xlsm,

the job runs successfully.

I

This made me believe that it has something to do with the tFTPPut filemask.

Aren't these ~$ files excluded by the RunIf expression already?

Should I tried to put the tFTP Put filemask in the RunIf trigger?

Thanks!

P.S. Here is the error message if the ~$All Supermarkets-TOTAL.xlsm file was in my local source folder:

(the job failed immediately)

[TRF]

As you are iterating after the tFileList component, you have to use (String)globalMap.get("tFileList_1_CURRENT_FILE") for the tFTPPut filemask.
Files will be transfered one at a time.

[Anonymous]

Hi TRF,

You are correct -- I didn't do that!  Though I put the RunIf condition in the trigger to exclude the ~$ files, but all is done in air. I didn't change the source directory in tFTPPut, the tFTPPut is still pulling all the files, and it doesn't go well with "~$" files.

Before your published your solution, I just used tFileCopy after the RunIf trigger to save the good files into another folder. After that, I do my tFTPPut with old filemasks. It worked.

But yours are better - with tFileList, I don't need to create a new folder to hold these good files.

Thank you!