topic Re: tmap temp files size is very huge issue. in Talend Studio

tmap temp files size is very huge issue.

Anonymous — Sat, 16 Nov 2024 12:24:38 GMT

Hi,
After execution the complex job in talend etl tool, we got the below are the tmap and lookup temp files

EQX_SDE_ORA_SalesOrderLinesFact_tMapData_Svra5G_row1_TEMP_0 - 871,658 KB BIN format
EQX_SDE_ORA_SalesOrderLinesFact_tMapData_Svra5G_row1_TEMP_1 - 871,650 KB BIN format
EQX_SDE_ORA_SalesOrderLinesFact_tMapData_Svra5G_Lookup_row18_ValuesData_0 - 12,138 KB
EQX_SDE_ORA_SalesOrderLinesFact_tMapData_Svra5G_Lookup_row18_KeysData_0 - 1,157 KB
..... number of lookup cache files.

clarifcation for :
EQX_SDE_ORA_SalesOrderLinesFact_tMapData_Svra5G_row1_TEMP_0 - 871,658 KB BIN format
EQX_SDE_ORA_SalesOrderLinesFact_tMapData_Svra5G_row1_TEMP_1 - 871,650 KB BIN format
why this tmap genrate the huge size generate the cache file? because this cache size file is very huge.
The LOOKUP cahe files generated size is good. but generating the huge size tmap cache file is advisable ot not?
what is the purpose of generating the tmap cahec files? how to avoid the tmap cahce file?
Thanks,
Srinivasan

Re: tmap temp files size is very huge issue.

Anonymous — Thu, 02 Feb 2012 08:17:47 GMT

Hi Srinivasan
This is because you check "Store temp data" true in tMap.
Files are huge and memory is limited. tMap has to save temp data into disk for calculation.
All these temp files will be deleted when the job complete.
Costing disk space for better performance is acceptable.
Regards,
Pedro

Re: tmap temp files size is very huge issue.

Anonymous — Thu, 02 Feb 2012 09:37:07 GMT

Hi Pedro,
Thanks for the explanation.
For improving the performance, we have given option "Store temp data" true in tMap.
the lookup table creating cache files size is enough good.
But in this tmap creating temp multiple file is very huge size and its too slow creating temp files.

I desinged job mentod:
1. Source and lookup table required column only.
2. used all the tmap component "store temp date" option to True.
3. Tmap Default buffer size : 2000000 . can i reduce the buffer size. if yes. how much should reduce the size?
4. Xms256M , Xmx1536M - memory setup.
Server configuration :
4GB RAM, WINDOWS mode 32 bit configuration.

please suggest me which are the methods to follow to improve the speed the tmap creating temp files(disk for calculation) ? or any other method?

Thanks,
Srinivasan S

Re: tmap temp files size is very huge issue.

Anonymous — Thu, 02 Feb 2012 09:55:37 GMT

Hi Srinivasan S
For No.1, No.2, No.4, I'm sure these methods will improve performance.
For No.3, this is related to current system situation(e.g memory).
Another advice is that you can try to reduce look-up times and optimize your job logic.
Regards,
Pedro

Re: tmap temp files size is very huge issue.

Anonymous — Fri, 03 Feb 2012 02:23:33 GMT

Hi Pedro,
Thanks for the valuable reply.
I tried to tune reduce lookup table data volume. still we facing temp file size is very huge. (the lookup table data volume around 1 billion in each lookup). in each lookup table used 3 column only.

do we have any compress option for creating the temp files? because temp file size going on very huge size (like 1GB of each files. totally created 6 files for the single job). also its takes so much time to creating the temp files.
we planning to execute concurrenlty all the jobs. (almost 20 jobs). The temp files size issue may occured in all the jobs.
please suggest me. how to proceeed to tune.

Thanks,
Srinivasan S

Re: tmap temp files size is very huge issue.

janhess — Fri, 03 Feb 2012 09:34:36 GMT

Have you got the inner joins set up correctly?

Re: tmap temp files size is very huge issue.

Anonymous — Fri, 03 Feb 2012 09:36:31 GMT

Hi Srinivasan S
Compressing temp files of tMap will have a bad effect on performance.
It's impossible to get perfect performace without space costing.
Regards,
Pedro

Re: tmap temp files size is very huge issue.

Anonymous — Mon, 06 Feb 2012 02:26:48 GMT

Thanks pedro. we are trying to improve the business logic level for tunning.
will let u know still facing performance issue.

Thanks,
Srinivasan S

Re: tmap temp files size is very huge issue.

Anonymous — Mon, 15 Oct 2012 18:57:35 GMT

HI, I have a issue with the temp file on the disk, when i finish the job the temp files in the folder are still there, and the whole temp folder is huge. Is this normal?
Thank you.

Re: tmap temp files size is very huge issue.

Anonymous — Tue, 16 Oct 2012 02:18:23 GMT

HI, I have a issue with the temp file on the disk, when i finish the job the temp files in the folder are still there, and the whole temp folder is huge. Is this normal?
Thank you.

Which version are you using? Can you please report a new topic for your problem?