topic Re: tWaitforFile triggers too soon in Talend Studio

tWaitforFile triggers too soon

_AnonymousUser — Sat, 16 Nov 2024 14:11:00 GMT

I need to use the trigger tWaitforFile component and then begin processing that file. The tWaitforFile component doesn't wait for the file to complete the transmission and prematurely begins the next step. tSleep won't help me because the file can be as many as 80 million rows and take an unknown amount of time to finish copying. How can I tell the component to trigger after file is finished?
- Bob

Re: tWaitforFile triggers too soon

Anonymous — Mon, 27 Oct 2008 13:49:49 GMT

Hello
The tWaitforFile is used to minitor a folder, if there is a new file created in this folder, it will tigger another action. on the next step, you can do your transmission. Note that, you should run the job first to monitor the folder.
Best regards
shong

Re: tWaitforFile triggers too soon

_AnonymousUser — Mon, 27 Oct 2008 14:03:52 GMT

Maybe I wasn't clear. I need to use the file in the next step, but it isn't done getting to the folder. How do I make Talend wait until the file is copied completely?

Re: tWaitforFile triggers too soon

Anonymous — Mon, 27 Oct 2008 16:17:12 GMT

you have to use the tSleep component after your tWaitForFile component, and set it to sleep for as long as it takes for the file to get fully loaded

Re: tWaitforFile triggers too soon

_AnonymousUser — Mon, 27 Oct 2008 16:29:58 GMT

But I don't know how long to make it sleep. Is there a way to detect if it is done?

Re: tWaitforFile triggers too soon

Anonymous — Mon, 27 Oct 2008 18:41:01 GMT

Not that I know of...
5484
maybe you could link to a tFileProperties and filter on a certain file size, but then you'd have to loop through (tWaitForFile -> tFileProperties -> tFilterRow) until the file was the size you wanted, so that might not be so helpful

Re: tWaitforFile triggers too soon

_AnonymousUser — Mon, 27 Oct 2008 19:39:19 GMT

Thank you for opening an issue log. I am sure I am not the only one who would like to see this. I will try the file properties loop as you suggest and hope for an easier component someday.

Re: tWaitforFile triggers too soon

Anonymous — Mon, 27 Oct 2008 20:07:24 GMT

or maybe you could write a code routine, or use one of the tJava* components and use one of these:
http://java.sun.com/javase/6/docs/api/java/nio/channels/FileChannel.html#tryLock()
http://java.sun.com/javase/6/docs/api/java/nio/channels/FileLock.html

Re: tWaitforFile triggers too soon

Anonymous — Wed, 03 Dec 2008 13:49:54 GMT

Bob,
I had the same problem as you, maybe it will help you.
I use tWaitForFile to wait until my report is generated and once it is ready I send it as attachment in e-mail (tSendMail).
I added 1 extra line at the very bottom of script where I generate report to copy created file into the same folder.
I rebuilt my job and I have now:
tWaitForFile_1 --iterate--> tWaitForFile_2 --iterate--> tSendMail_1 --onComponentOk--> tFileDelete_1
In tFileDelete_1 I delete copied file (from tWaitForFile_2).
I hope it helps for you.
Marcin

Re: tWaitforFile triggers too soon

Anonymous — Wed, 03 Dec 2008 13:52:13 GMT

strpdsnk,
your solution looks more profesional.
Could you provide few lines of code with how it should be written for people who are not experienced in Java, please?
Thanks,
Marcin

Re: tWaitforFile triggers too soon

Anonymous — Thu, 04 Dec 2008 17:49:22 GMT

Bob,
There's always the "old way." I had a similar problem and had the people who send the file add a final step to THEIR job that generates the file to create a second file called "DONE" in the same directory. So, then all I have to do is look for DONE file. Because it only shows up when the sender job has completed writing the file.

Re: tWaitforFile triggers too soon

Anonymous — Mon, 23 Mar 2009 22:13:34 GMT

Is there a way to specify a quiesce time? ie. where the current time - last time the file was accessed/modified >= quiesce time. Can i open a request or bug on this somewhere?

Re: tWaitforFile triggers too soon

_AnonymousUser — Mon, 23 Mar 2009 23:48:30 GMT

Since our file transfer could be anywhere from 5 seconds to 5 hours, we needed a solution that was checking the file. We didn't want to check the size changing because we felt this may be unreliable and we needed a 100% failure proof solution. Our solution to this was to use the tJavaFlex component and pur this code into the "start" code area. Use this component following the tWaitforFile.
CheckFileUsage FileChecker = new CheckFileUsage();
boolean fileInUse = true;
context.customerFileIn = ((String)globalMap.get("tWaitForFile_1_FILENAME"));

while (fileInUse) {
try{FileChecker.FileUseCheck (context.nasFolderIn + context.customerFileIn);
fileInUse = false; }
catch (IOException e) {System.err.println("ERROR TRAP");
System.err.println (e);}
}
- Bob

Re: tWaitforFile triggers too soon

_AnonymousUser — Mon, 23 Mar 2009 23:51:59 GMT

Oh yeah....
this line --->> try{FileChecker.FileUseCheck (context.nasFolderIn + context.customerFileIn);
should have your file name in it.
- Bob

Re: tWaitforFile triggers too soon

Anonymous — Fri, 09 Aug 2019 10:22:45 GMT

Hi,

I faced this same issue and there is quick fix available in talend that you can do to resolve this issue of partial file getting loaded while the complete file is not transmitted.

In tWaitForFile component, go to Advance settings, tick the checkbox for 'Wait for file to be released' and provide the count of milliseconds you want to wait for this check to be done on your file like every 'X' number of ms, component will check if file is released from any other operation, if yes, then it moves to next component else it continues to wait till the complete file is released.

tWait_for_File_release_settings.PNG