topic Re: [resolved] Using tSCPPut component in Talend Studio

[resolved] Using tSCPPut component

Anonymous — Tue, 14 May 2013 14:00:23 GMT

Hi,
I have a list of files with a certain extension (say .abc) under my local system in the directory (C:\tmp\folder1\one.abc, c:\tmp\folder2\two.abc) and so on..
I want to transfer each such folder into a remote machine using the SCP components (tSCPPut to be precise). What I have observed is that if I have a folder called "/root/temp/" as the Remote Directory that exists inside the remote machine, only the files are put one after another (I am fetching list of files through tFileLIst).
However if I need to put them into a non-existent directory, say "/root/temp/output/", I am unable to do so, and the error "Caused by: java.io.IOException: Remote scp terminated with error (scp: /root/temp/output/: No such file or directory)." pops up.
This means that a non existent directory is not created automatically during copying of the files.
Can anyone guide me to an alternative to this? How can I make use of the tJava component to remotely create a directory and then copy the files?
I am looking to copy the files into remote locations "/root/temp/output/folder1/one.abc", "/root/temp/output/folder2/two.abc" and so on.

Re: [resolved] Using tSCPPut component

Anonymous — Tue, 14 May 2013 21:16:30 GMT

Indeed this option to create a remote directory is highly missing. As long as we have that, you could check if the remote directory exists with tSCPFileExists.
Use the run if connector and set as condition:
"File exists.".equals((String)globalMap.get("tSCPFileExists_1_STATUS")) == false
Yes I know, this looks silly, but unfortunately this component is not very well designed.
This small job shows checks if the directory "new_dir" under the user home exists and if not it will be created.

Re: [resolved] Using tSCPPut component

Anonymous — Wed, 15 May 2013 13:42:09 GMT

Thank you very much, this helps a lot!
Another question I had was, say in this case, if I want to create "new_dir/tmp/folder1" directory, where "tmp" directory doesn't exist, then even "folder1" directory won't get created right?
So I will need to follow the hierarchy and create directories as I go along, and then copy the files?

Re: [resolved] Using tSCPPut component

Anonymous — Wed, 15 May 2013 21:33:29 GMT

Unfortunately your are right. You have to follow from the root to the leaf. Typically I build that in a component and I think it would be a great idea to add this function into the tSCPPut component.
Hint: I would start wit a check if your target structure exists and go back to the root. It is often the situation, that only one or two of the last directories are missing.

Re: [resolved] Using tSCPPut component

Anonymous — Thu, 16 May 2013 09:36:06 GMT

Yes, you're correct, it usually is for creating one or two leaf directories.
I found a command to create such directories that do not exist. We can use "mkdir -p root/temp/new_dir/folder1". What this will do is, create folder1, new_dir and temp folders, if it doesn't already exist.
Thanks again for your prompt reply and help!

Re: [resolved] Using tSCPPut component

Anonymous — Thu, 16 May 2013 20:18:20 GMT

Thanks a lot, I will use your hint about the mkdir option.

Re: [resolved] Using tSCPPut component

_AnonymousUser — Wed, 17 Sep 2014 21:38:13 GMT

You can call "mkdir -p" and all the subdirectories will automatically be made.