I need to use the trigger tWaitforFile component and then begin processing that file. The tWaitforFile component doesn't wait for the file to complete the transmission and prematurely begins the next step. tSleep won't help me because the file can be as many as 80 million rows and take an unknown amount of time to finish copying. How can I tell the component to trigger after file is finished?
- Bob
Bob, There's always the "old way." I had a similar problem and had the people who send the file add a final step to THEIR job that generates the file to create a second file called "DONE" in the same directory. So, then all I have to do is look for DONE file. Because it only shows up when the sender job has completed writing the file.
Is there a way to specify a quiesce time? ie. where the current time - last time the file was accessed/modified >= quiesce time. Can i open a request or bug on this somewhere?
Since our file transfer could be anywhere from 5 seconds to 5 hours, we needed a solution that was checking the file. We didn't want to check the size changing because we felt this may be unreliable and we needed a 100% failure proof solution. Our solution to this was to use the tJavaFlex component and pur this code into the "start" code area. Use this component following the tWaitforFile.
CheckFileUsage FileChecker = new CheckFileUsage();
boolean fileInUse = true;
context.customerFileIn = ((String)globalMap.get("tWaitForFile_1_FILENAME"));
I faced this same issue and there is quick fix available in talend that you can do to resolve this issue of partial file getting loaded while the complete file is not transmitted.
In tWaitForFile component, go to Advance settings, tick the checkbox for 'Wait for file to be released' and provide the count of milliseconds you want to wait for this check to be done on your file like every 'X' number of ms, component will check if file is released from any other operation, if yes, then it moves to next component else it continues to wait till the complete file is released.
