tFileOutputDelimited Lock until Complete ?

[DarinAfni]

Hello,

I have a simple process that writes a delimited file from a database table to a network directory.  The target directory has a different 3rd party software program that runs on a schedule to check for new files in this target directory and subsequently sends those files to a vendor for processing.

The issue that I have is that the file I'm building is over 9 million records and almost 3 GB.  The vendor requires this file to be sent in a single file rather than chunking it into multiple files.  My file build process doesn't complete before the schedule runs to send files and it's picking up this partially built file before my process has completed the file build and breaking my subsequent processes.

Is there a standard procedure that I'm ignorant to that I should be doing to keep a read lock on the tFileOutputDelimited output from being read by other users until that lock is removed?  I don't see anything in the advanced settings that would correspond to this.  My only other approach is to build the file in a staging directory and then just do a copy to the actual target and cross my fingers that the same issue doesn't persist.

Are there any pointers for what I think must be a fairly common issue?

Thanks,

Darin

[rmartin2]

Hi !

The standard procedure is to name your file like "my_file.part" while copying data, and then just rename it with a tCopy or tSystem (both works), to "my_file.zip" when finished.

You're program should not be able to take the .part file, and renaming is an atomic instruction without delay

Sincerely,

[DarinAfni · Accepted solution]

Thanks rmartin.

I ended up writing the file to a staging/working directory and then used a tSystem component to move the file to the target directory (I find the OS move to be faster than using the tFileCopy component with large files).

The .part wasn't sufficient enough to keep the write process atomic.