tMap: Loads very slow with big schema as input

Anonymous — Mon, 28 Nov 2016 12:47:45 GMT

Hello,

i am currently working on a Job that imports a bigger fileDelimited with multiple hundreds of columns.
The creation of the fileDelimited as metadata needs about 15 minutes to read the schema. That is ok beacuse its mostly a one time task but every "tMap" i open (that has the complete file-schema as input.row) needs more than 5 minutes to load. Also finding the correct column in a dropdown list (on other components) where i can not search is pretty time consuming.
Is there a better way to work with this? I need the tMap for Joins (maybe this can be replaced by "tJoin" to not load all columns on editing?) but i still need the tMap to map the input columns to the correct output and transform them as necessary.
I already increased the memory limit for Talend Open Studio but this is not the limiting factor. The CPU instead is at 99% while opening a tMap.

Can anyone help me with this? Currently i spend most of the time waiting for a tMap to open^^.

Re: tMap: Loads very slow with big schema as input

Anonymous — Mon, 05 Dec 2016 03:15:09 GMT

Hi,
Have you tried to check out " Temp data directory path" option on tMap component to s tore the temporary data generated for lookup loading to see if it is ok with you?
Could you please post your job setting screenshots into forum so that we can get more information about your job.
In addition, talend provides user with dynamic schema which allows you to design schema with an unknown column structure (unknown name and number of columns).

Best regards
Sabrina

topic Re: tMap: Loads very slow with big schema as input in Talend Studio

tMap: Loads very slow with big schema as input

Re: tMap: Loads very slow with big schema as input