[resolved] Using of tSCPget, tSCPdelete with special conditions in a iteration
Hi all,
I'm using TIS 4.2 and I hope that someone can help me in following scenario:
There are 3 Server, remote server1, remote server2 and local server3.
The remote server1 sends a unknown number of data files to remote Server2.
When every sending of one of these data file is completed, the server1 sends a file to server2 with the same name of the
sent data file added with the filename-suffix "ok". These ok-files has the size of 0 Byte.
(Example: sending DataFile1 - sending DataFile1.ok - sending DataFile2 - sending DataFile2.ok ....).
The Filename of DataFile.. is unknown, but it exists a known Filemask which has constant and variable parts.
My problems are following:
1. I don't know when sending from server1 to server2 ist completed.
2. I have to get all the data Files which also have a existing corresponding 0 Byte *.ok-File from the remote Server2 to my local server3.
For this I have a tSCPConnection between my local server3 and the remote server2 succesfully established. But, if a corresponding *.ok-File don't exist on server2, I have to do nothing with the existing data file.
3. After this, when the receiving of every data file to the local server3 is completed I have to delete
the received datafile AND the corresponding *.ok-File on the remote server2.
I hope that someone can help me and has a easy solution for the described scenario because I don't know how i can use tSCPlist, tSCPexists, tSCPget and tSCPdelete in a iteration with these conditions.
Many thanks.
Regards Detlef
Hi
Add a filter expression in the command to only list the files with suffix .ok, for example:
"cd /the directory path/;ls *.ok"
and then, parse the file name and get the corresponding file without suffix, which will be deleted as well. The job looks like:
tSCPFileList_1--iterate--tJava--oncomponentok---tSCPGet--oncomponentok-->tSCPDelete_1--oncomponentok--->tSCPDelete_2
|
onsubjobok
|
tFileList--iterate-->SCPPUT
on tJava:
String current_line=((String)globalMap.get("tSCPFileList_1_CURRENT_LINE")); //current file name, for example:EFGH.2222.X3333.ok
String sourceFile=current_line.substring(0,current_line.lastIndexOf(".")); //get its corresponding file without suffix, for example:EFGH.2222.X3333
globalMap.put("sourceFile", sourceFile);
on tSCPDelete_1: delete current file name on server2, set the file list as "the directory path"+(String)globalMap.get("tSCPFileList_1_CURRENT_LINE")
on tSCPDelete_2: delete its corresponding source file, set the file list as "the directory path"+(String)globalMap.get("sourceFile")
1. I don't know when sending from server1 to server2 ist completed.
How do you send the files from server1 to server2? Using Talend components or other way? In talend job, the next processing will be triggered with the connectors like 'onsubjobok', 'oncomponentok' once it finishes.
The job design of sending files from server2 to server 3 look like:
tSCPFileList_1--iterate--tSCPGet--oncomponentok-->tSCPDelete
|
onsubjobok
|
tFileList--iterate-->SCPPUT
on tSCPFileList_1: iterate each file on server2, set the filelist as "cd /the directory path/;ls -1"
on tSCPGet: get each file into local directory, set the filelist as "the directory path"+(String)globalMap.get("tSCPFileList_1_CURRENT_LINE")
on tSCPDelete: delete the file from server2 once it is downloaded, set the file list as "the directory path"+(String)globalMap.get("tSCPFileList_1_CURRENT_LINE")
on tFileList: iterate each file in local directory
on tSCPPut: put each file to server3, set the filelist as ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))
Best regards
Shong
Hi Shong, many thanks for your fast reply,
but, I'm sorry, it don't works good for me.
First your question: The remote server1 sends files to remote server2 and this is without using Talend components (unknown other way).
It is not a problem for me to get all files from remote server2 to local server3 and to delete all after receiving
(tSCPFileList_1--iterate--tSCPGet--oncomponentok-->tSCPDelete)
but i'am looking for a solution for the described situations which are following:
On remote server2 you can find 2 example situations:
1. Existing files:
"ABCD.1234.X5678" - Size for example 1200 byte (File with Data)
"ABCD.1234.X5678.ok" - Size 0 byte (empty File without Data)
"EFGH.2222.X3333" - Size for example 2500 byte (File with Data)
"EFGH.2222.X3333.ok" - Size 0 byte (empty File without Data)
2. Existing files:
"ABCD.1234.X5678" - Size 0 byte, maybe increasing in time
"EFGH.2222.X3333" - Size for example 2500 byte (File with Data)
"EFGH.2222.X3333.ok" - Size 0 byte (empty File without Data)
The difference between situation 1 and 2 is the missing of the *.ok-File corresponding to file "ABCD.1234.X5678" .
What should the program to do now?
In situation 1 I would like to receive all data files (only them) to local server3. The *.ok-Files I do'nt want to receive.
After that i want to delete all 4 files on remote server2.
In situation 2 I only want to receive the file "EFGH.2222.X3333" to local server3.
The coresponding *.ok-File I don't want to receive.
The file "ABCD.1234.X5678" i also don't want to receive, because there is no file with the name "ABCD.1234.X5678.ok".
After that i want to delete the files "EFGH.2222.X3333" and "EFGH.2222.X3333.ok" on remote server2.
The file "ABCD.1234.X5678" I don't want to delete now on the remote server2, because the file is not received
I hope so that you, or anyone in the forum, have a solution for the described problem.
With best regards
Detlef
You need to get a file list of the ok files with tSCPFileList. For each file you need to do tSCPGet of the data file by using the file name of the ok file minus the '.ok' then do the tSCPDelete.
Hi janhess,
this is very interesting,
but how can i get a file list with the *.ok-files. In tSCPFileList there is no way to work with filemasks like "*.ok".
An other question by the way, how can I separate the suffix and praefix of a known filename?
with regards
Detlef
Hi
Add a filter expression in the command to only list the files with suffix .ok, for example:
"cd /the directory path/;ls *.ok"
and then, parse the file name and get the corresponding file without suffix, which will be deleted as well. The job looks like:
tSCPFileList_1--iterate--tJava--oncomponentok---tSCPGet--oncomponentok-->tSCPDelete_1--oncomponentok--->tSCPDelete_2
|
onsubjobok
|
tFileList--iterate-->SCPPUT
on tJava:
String current_line=((String)globalMap.get("tSCPFileList_1_CURRENT_LINE")); //current file name, for example:EFGH.2222.X3333.ok
String sourceFile=current_line.substring(0,current_line.lastIndexOf(".")); //get its corresponding file without suffix, for example:EFGH.2222.X3333
globalMap.put("sourceFile", sourceFile);
on tSCPDelete_1: delete current file name on server2, set the file list as "the directory path"+(String)globalMap.get("tSCPFileList_1_CURRENT_LINE")
on tSCPDelete_2: delete its corresponding source file, set the file list as "the directory path"+(String)globalMap.get("sourceFile")
