tMap - Is there a way to force the Lookup's temp files to store on disk after job runs?
Our reference file has over 100M rows of data.
The job compares a file with new data against the reference file and rejects the new data to a rejects file.
The tMap stores the reference file's data to a temp folder on disk before the lookup is done.
How can I force Talend DI to keep the reference data in the .bin files in the temp folder on disk after the job finishes running, so the job runs quicker next time? Currently, the files in the temp folder are deleted when the job is done and they have to be recreated the next time the job runs.
You cannot store the temp data permanently as it will beat the original requirement to clear the temp space after processing is complete. Since temp data is also a file, there will be still file I/O operations which you cannot avoid. Another flip side of your approach is the overhead and file management issues when the lookup file gets modified.
Considering the file size and processing need, why dont you do this operation using Bigdata Spark Batch job? It will be much faster for these type of huge file operations.
