HELP! Very slow in generating the data into a shared folder location

[Anonymous]

Hi,

I hope everyone is all well.

I am newbie in Talend application. I created the job below to extract 60-100k or more records which I would like to directly save in a network shared folder location. However, it takes a long time to generate (half an hour) and finish the job that is why I always "kill" the running session. If I ran it pointing to my home folder or even running in the Talend Cloud, it just took about 5 minutes or less. Is there any other way or I additional component that I need to include to get my Talend job optimized and run faster? Any help is greatly appreciated. Thank you.

[Anonymous]

It sounds weird, the error does not occur if you don't use a tFileCopy in the job? but the error thrown on tSalesforceInput.

[Anonymous]

Correct. It is weird. It throws me the said error if I enabled the tFilecopy component.

Here is the error:

 

I'm thinking if there is any other design that I could apply like archiving first before transferring or copying the file onto the shared folder.

[Anonymous]

Hi @Shicong Hong​ ,

It is now working and running without having any errors so far. However, it's been almost 5 hours and it's still running. Though, I can see that it is now generating the file and the size is getting bigger. Is there any other option that I can apply in the job to speed up the process?

Thank you much in advance.

[Anonymous]

Hi @guenneguez jeremy​ ,

Hi @Shicong Hong​ ,

It is now working and running without having any errors so far. However, it's been almost 5 hours and it's still running. Though, I can see that it is now generating the csv file and the size is getting bigger. Is there any other option that I can apply in the job to speed up the process?

Thank you much in advance.

[Anonymous]

@Not defined Not defined​, try the following points to optimize the job:

1 Allocate more memory to the job execution.

2 Filter the data on tMap, don't need to use a tFilterRow.

3 Copy the file in next subjob after it is generated, so change the job like:

tSalesforceInput--main--tConvertType--main--tMap--tFileOutputDelimited

|onsubjobok

tFileCopy

Hope it helps you!

Regards

Shong

[Anonymous]

Thank you, @Shicong Hong​ .

I applied your recommendations.

• Increased the memory allocation

• Removed the tFilterRow
• And on subjobok
• I set the Query Mode to "Query" which took more than 7hrs to run. However, If I changed it back to "Bulk" or "BulkV2" the error "connection reset" will appear. Do we have any other way to fix the connection reset to at least trim down the run time?

Thank you much.