topic Re: HELP! Very slow in generating the data into a shared folder location in Talend Studio

HELP! Very slow in generating the data into a shared folder location

Anonymous — Fri, 15 Nov 2024 23:08:16 GMT

Hi,

I hope everyone is all well.

I am newbie in Talend application. I created the job below to extract 60-100k or more records which I would like to directly save in a network shared folder location. However, it takes a long time to generate (half an hour) and finish the job that is why I always "kill" the running session. If I ran it pointing to my home folder or even running in the Talend Cloud, it just took about 5 minutes or less. Is there any other way or I additional component that I need to include to get my Talend job optimized and run faster? Any help is greatly appreciated. Thank you.

Re: HELP! Very slow in generating the data into a shared folder location

gjeremy1617088143 — Thu, 10 Mar 2022 08:11:40 GMT

Hi, maybe you could send the result in an Output Stream, tFileOutputDelimited can do this, and after that you transform your OutputStream into a FileOutputStream and write it on you shared folder(necessite to write a bit a of java).

So the only action on your shared folder will be to just write once the file.

Re: HELP! Very slow in generating the data into a shared folder location

Anonymous — Thu, 10 Mar 2022 08:32:02 GMT

I would suggest the following ways to optimize the performance.

- Check the manual query box, write custom query and add condition in the query to filter the records on tSalesforceInput.

- Allocate more memory to job execution.

- Store the records into local file, and finally, copy the file to shared folder using tFileCopy.

Regards

Shong

Re: HELP! Very slow in generating the data into a shared folder location

gjeremy1617088143 — Thu, 10 Mar 2022 08:34:21 GMT

eg in a tJava at the begining of the job :

in basic parameter :

ByteArrayOutputStream stream = new ByteArrayOutputStream();

globalMap.put("outStream",stream);

in advance parameter :

import java.io.ByteArrayOutputStream;

in the tFileOutputDelimited check the use stream output checkbox

and in the stream name field :

(ByteArrayOutputStream)globalMap.get("outStream")

then in tJava at the end of the job :

in basic parameters :

FileOutputStream fos = new FileOutputStream("your file path");

fos.write(((ByteArrayOutputStream)globalMap.get("outStream")).toByArray());

fos.close;

in Advance Parameters :

import java.io.FileOutputStream;

Re: HELP! Very slow in generating the data into a shared folder location

Anonymous — Mon, 14 Mar 2022 05:41:22 GMT

Hi Shong,

Thank you. I followed your suggestion and tried to create first a copy to the local file and copy it to shared folder afterwards. However, I got a new error "Connection Reset". I've already adjusted the Timeouts from the Talend Studio but still the same.

Re: HELP! Very slow in generating the data into a shared folder location

Anonymous — Tue, 15 Mar 2022 08:11:00 GMT

Hello,

Thank you much for your recommendation. 🙂

I tried adding the tJava (start and end of the job) and its codes, but I'm getting the error below.

Re: HELP! Very slow in generating the data into a shared folder location

Anonymous — Tue, 15 Mar 2022 08:30:53 GMT

@Not defined Not defined any changes you have made on tSalesforceInput? use manual query? Have you got this error before?

Re: HELP! Very slow in generating the data into a shared folder location

Anonymous — Tue, 15 Mar 2022 08:32:23 GMT

I see you are using runIf trigger, make sure you have set the condition expression (boolean type).

Re: HELP! Very slow in generating the data into a shared folder location

Anonymous — Tue, 15 Mar 2022 08:45:28 GMT

Yes, I've used the manual query and tried as well the Built-In schema but still the same error. The error occurs if I added the tFileCopy component which is copying the file from local to the remote server shared destination.

Re: HELP! Very slow in generating the data into a shared folder location

gjeremy1617088143 — Tue, 15 Mar 2022 09:01:35 GMT

could you go to the code view, click on the red square and send a screenshot of the code ?

Re: HELP! Very slow in generating the data into a shared folder location

Anonymous — Tue, 15 Mar 2022 09:26:37 GMT

It sounds weird, the error does not occur if you don't use a tFileCopy in the job? but the error thrown on tSalesforceInput.

Re: HELP! Very slow in generating the data into a shared folder location

Anonymous — Mon, 21 Mar 2022 05:43:16 GMT

Correct. It is weird. It throws me the said error if I enabled the tFilecopy component.

Here is the error:

I'm thinking if there is any other design that I could apply like archiving first before transferring or copying the file onto the shared folder.

Re: HELP! Very slow in generating the data into a shared folder location

Anonymous — Wed, 23 Mar 2022 05:47:05 GMT

Hi @Shicong Hong ,

It is now working and running without having any errors so far. However, it's been almost 5 hours and it's still running. Though, I can see that it is now generating the file and the size is getting bigger. Is there any other option that I can apply in the job to speed up the process?

Thank you much in advance.

Re: HELP! Very slow in generating the data into a shared folder location

Anonymous — Wed, 23 Mar 2022 05:48:13 GMT

Hi @guenneguez jeremy ,

Hi @Shicong Hong ,

It is now working and running without having any errors so far. However, it's been almost 5 hours and it's still running. Though, I can see that it is now generating the csv file and the size is getting bigger. Is there any other option that I can apply in the job to speed up the process?

Thank you much in advance.

Re: HELP! Very slow in generating the data into a shared folder location

Anonymous — Wed, 23 Mar 2022 07:19:08 GMT

@Not defined Not defined, try the following points to optimize the job:

1 Allocate more memory to the job execution.

2 Filter the data on tMap, don't need to use a tFilterRow.

3 Copy the file in next subjob after it is generated, so change the job like:

tSalesforceInput--main--tConvertType--main--tMap--tFileOutputDelimited

|onsubjobok

tFileCopy

Hope it helps you!

Regards

Shong

Re: HELP! Very slow in generating the data into a shared folder location

Anonymous — Mon, 28 Mar 2022 09:49:16 GMT

Thank you, @Shicong Hong .

I applied your recommendations.

Increased the memory allocation

Removed the tFilterRow
And on subjobok
I set the Query Mode to "Query" which took more than 7hrs to run. However, If I changed it back to "Bulk" or "BulkV2" the error "connection reset" will appear. Do we have any other way to fix the connection reset to at least trim down the run time?

Thank you much.