topic Re: [resolved] File sync in Talend Studio

[resolved] File sync

_AnonymousUser — Sat, 16 Nov 2024 12:29:44 GMT

Hi I want to keep the files at two locations in sync with each other.The two file locations will be passed as the context variables.
e.g.
let one location is C:/source
and the other is C:/target
When my job runs for the first time it should first check the source folder and match them with the contents of the target folder. If there is a difference in the two locations the files need to be copied to the other location.Basically the two folders need to be in sync with each other. If there are new files in the target folder they need to be copied to the source and vice versa.

Any help on the above will be highly appreciated.
Thanks in Advance
Abhineet

Re: [resolved] File sync

Anonymous — Wed, 21 Dec 2011 04:37:39 GMT

Hi
Using tFileList to iterate each file in source folder, check the file if it exits in the target using tFileExist, copy it to target folder with tFileCopy if not. Foe example:
tFileList--iterate--tJava--componentOK--tFileExist--runIf--tFileCopy
Best regards
Shong

Re: [resolved] File sync

_AnonymousUser — Wed, 21 Dec 2011 04:50:59 GMT

Thanks Shong!
Seems good..Do I need to follow the same steps in another flow while I check the target folder?
Because there may be a scenario that there is a new file in the target folder that also needs to be copied in the source folder.
Regards,
Abhineet

Re: [resolved] File sync

Anonymous — Wed, 21 Dec 2011 06:05:52 GMT

Hi
Yes, use the same way to check the target folder in next subjob.
Best regards
Shong

Re: [resolved] File sync

_AnonymousUser — Wed, 21 Dec 2011 10:30:29 GMT

Hi
Using tFileList to iterate each file in source folder, check the file if it exits in the target using tFileExist, copy it to target folder with tFileCopy if not. Foe example:
tFileList--iterate--tJava--componentOK--tFileExist--runIf--tFileCopy
Best regards
Shong

Hi I trying the same solution but I am bit unfamiliar with Java.Can u help me out with the sample Java code to be used in tJava component.
Regards,
Abhineet

Re: [resolved] File sync

Anonymous — Wed, 21 Dec 2011 12:08:22 GMT

Hi,
Don't think you need to use a tJava (shong correct me if i'm wrong ). Just link your tfileList to the tFileExist.
tFileList--iterate--tfileExist---runif---tfilecopy

Re: [resolved] File sync

_AnonymousUser — Wed, 21 Dec 2011 12:31:41 GMT

Hi,
Don't think you need to use a tJava (shong correct me if i'm wrong ). Just link your tfileList to the tFileExist.
tFileList--iterate--tfileExist---runif---tfilecopy

I'm not sure if tfileExist is a standalone component because when I am using the combination
tFileList--iterate--tfileExist---runif---tfilecopy
it gives me a syntax error in the tfileExist component.
Regards,
Abhineet

Re: [resolved] File sync

Anonymous — Wed, 21 Dec 2011 12:54:58 GMT

show me your job and what you have put in your tfileexist component

Re: [resolved] File sync

_AnonymousUser — Wed, 21 Dec 2011 13:00:25 GMT

My Bad!
seems to be working now...guess there was some mistake in the "if" condition.

Regards,
Abhineet

Re: [resolved] File sync

Anonymous — Wed, 21 Dec 2011 13:18:10 GMT

Sometimes it's good to switch in the source view, it's easier to detect the errors

Re: [resolved] File sync

Anonymous — Wed, 21 Dec 2011 13:22:41 GMT

Hi
@Or3l: I use tJava to get the current file name and initialize variables, such as the target file path, which will be used on tFileExist.
Best regards
Shong.

Re: [resolved] File sync

_AnonymousUser — Wed, 21 Dec 2011 13:31:32 GMT

Finally it worked that too without a tJava
Thanks Guys!!
Regards,
Abhineet

Re: [resolved] File sync

_AnonymousUser — Thu, 22 Dec 2011 05:00:29 GMT

Hi,
appologies to reopen the resolved item.
As a part of the same requirement i need to copy only those files which are created not more the a minute before.
I was thinking of using a tfileProperties along with the below flow

tFileList--iterate--tfileExist---runif---tfilecopy
|
|
iterate(order2)
|
tfileProperties
and give the condition in runIf , something like (systemTime - filemodifiedTime < 1).
please suggest if it is the correct approach or can be achieved in some other way.

Regards,
Abhineet

Re: [resolved] File sync

_AnonymousUser — Thu, 22 Dec 2011 05:02:22 GMT

Hi,
appologies to reopen the resolved item.
As a part of the same requirement i need to copy only those files which are created not more the a minute before.
I was thinking of using a tfileProperties along with the below flow

tFileList--iterate--tfileExist---runif---tfilecopy
|
|
iterate(order2)
|
tfileProperties
and give the condition in runIf , something like (systemTime - filemodifiedTime < 1).
please suggest if it is the correct approach or can be achieved in some other way.

Also , if the above approach is correct how to achieve that.
Regards,
Abhineet

Re: [resolved] File sync

_AnonymousUser — Thu, 22 Dec 2011 06:12:01 GMT

Hi Team,
Can somebody help me on that?
Regards,
Abhineet

Re: [resolved] File sync

Anonymous — Thu, 22 Dec 2011 08:34:50 GMT

Hi,
I think I would first get the file that have been created not more than a minute before, then check if there are already in the target folder and then copy them. So It would give sthing like this
tFileList--iterate---tFileProperties----runif----tfileexist---runif---tfilecopy
One thing toknow is that the mtime property is based on the datetime modification of the file, so it's not really the date of creation of the file.
Furthermore I don't know if you have already managed that but you should probably "tag" the files you have already synchronized (by adding something in their name for example). Unless your job will be quite long.

Re: [resolved] File sync

_AnonymousUser — Thu, 22 Dec 2011 09:18:53 GMT

Hi,
I tried the above connection but it seems that the connection "tFileProperties----runif----tfileexist" is not feasible. It gives me a warning that tfileProperties must have an output attached.
Regards,
Abhineet

Re: [resolved] File sync

ytuzet — Thu, 22 Dec 2011 09:31:30 GMT

Hi,
put a tflowtoiterate between tfileproperties an the runif

Re: [resolved] File sync

Anonymous — Mon, 28 Dec 2015 12:58:47 GMT

Use tfilelist---iterate-->tFilecopy
in tfilecopy you can use
filename ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))
Replace Existing files checked