HELP! Very slow in generating the data into a shared folder location

[Anonymous]

Hi,

I hope everyone is all well.

I am newbie in Talend application. I created the job below to extract 60-100k or more records which I would like to directly save in a network shared folder location. However, it takes a long time to generate (half an hour) and finish the job that is why I always "kill" the running session. If I ran it pointing to my home folder or even running in the Talend Cloud, it just took about 5 minutes or less. Is there any other way or I additional component that I need to include to get my Talend job optimized and run faster? Any help is greatly appreciated. Thank you.

[gjeremy1617088143]

Hi, maybe you could send the result in an Output Stream, tFileOutputDelimited can do this, and after that you transform your OutputStream into a FileOutputStream and write it on you shared folder(necessite to write a bit a of java).

So the only action on your shared folder will be to just write once the file.

[Anonymous]

Hi

I would suggest the following ways to optimize the performance.

- Check the manual query box, write custom query and add condition in the query to filter the records on tSalesforceInput.

- Allocate more memory to job execution.

- Store the records into local file, and finally, copy the file to shared folder using tFileCopy.

Regards

Shong

[gjeremy1617088143]

eg in a tJava at the begining of the job :

in basic parameter :

ByteArrayOutputStream stream = new ByteArrayOutputStream();

globalMap.put("outStream",stream);

in advance parameter :

import java.io.ByteArrayOutputStream;

in the tFileOutputDelimited check the use stream output checkbox

and in the stream name field :

(ByteArrayOutputStream)globalMap.get("outStream")

then in tJava at the end of the job :

in basic parameters :

FileOutputStream fos = new FileOutputStream("your file path");

fos.write(((ByteArrayOutputStream)globalMap.get("outStream")).toByArray());

fos.close;

in Advance Parameters :

import java.io.FileOutputStream;

[Anonymous]

Hi Shong,

Thank you. I followed your suggestion and tried to create first a copy to the local file and copy it to shared folder afterwards. However, I got a new error "Connection Reset". I've already adjusted the Timeouts from the Talend Studio but still the same.

 

 

[Anonymous]

Hello,

Thank you much for your recommendation. 🙂

I tried adding the tJava (start and end of the job) and its codes, but I'm getting the error below.

 

[Anonymous]

@Not defined Not defined​  any changes you have made on tSalesforceInput? use manual query? Have you got this error before?

[Anonymous]

I see you are using runIf trigger, make sure you have set the condition expression (boolean type).

[Anonymous]

Yes, I've used the manual query and tried as well the Built-In schema but still the same error. The error occurs if I added the tFileCopy component which is copying the file from local to the remote server shared destination.

[gjeremy1617088143]

could you go to the code view, click on the red square and send a screenshot of the code ?