topic Re: tPivotToColumnsDelimited is very slow in Talend Studio

tPivotToColumnsDelimited is very slow

Anonymous — Sat, 16 Nov 2024 11:04:08 GMT

Hi,
When i have tried loading some 1 million data using tPivotToColumnsDelimited it's taking around 7 hours.
I thing its not de normalizing when it receives any row.its collecting all the row first and de normalizing after collecting all.
Do you have any approach that will make the process faster.
My Scenario is:
I want to load about 1 month data(around 5 crores) from MongoDB to a table which has a de normalized format.
so i need to write it through a tPivotToColumnsDelimited file and then load it to the table.
is there any way that i can load all the one month data in 7 to 8 hours?
Is there any mechanism to use,so that it will use local-memory to store temporarily.
Any way will be appreciated....I need this urgently please.....as i am not find any way to make the load fast.
Thanks,
jyotiranjan

Re: tPivotToColumnsDelimited is very slow

Anonymous — Sun, 06 Sep 2015 05:09:45 GMT

Hi jyotiranjan,

Can you provide some screenshots of your current job? It will be helpful for us to locate the problem.

Best regards
Sabrina

Re: tPivotToColumnsDelimited is very slow

Anonymous — Mon, 14 Sep 2015 10:11:04 GMT

Hi jyotiranjan,

Can you provide some screenshots of your current job? It will be helpful for us to locate the problem.

Best regards
Sabrina

Hi Sabrina,
Please find below the screen shot of my job.

Here the taggregate is extracting a nested json value and copying into another file.
The tpivotcolumn.... is taking a lot of time

Re: tPivotToColumnsDelimited is very slow

Anonymous — Fri, 18 Sep 2015 04:54:30 GMT

Hi jyotiranjan,
Can you upload again the screenshots you wanted to show, please? For some reason it didn't make it to your post.

Best regards
Sabrina