topic Re: Job losses responsiveness in Talend Studio

Job losses responsiveness

KKlimczuk1618309742 — Sat, 16 Nov 2024 00:03:47 GMT

Hello,

I am running very simple job which loads data from couchbase and displays documents on console (screenshot attached). This job connect a bucket with 182k of documents, when I am running this job it start to print documents on console and when it reaches about 61k of documents it stops printing anything to logs and stops refreshing status on job components links.

I have tried to enable logging to file but no file has been created.

Re: Job losses responsiveness

gjeremy1617088143 — Mon, 14 Jun 2021 14:12:43 GMT

HI, have you tried to connect it to a tfileoutput delimited for example ?

Send me Love and Kudos

Re: Job losses responsiveness

Anonymous — Tue, 15 Jun 2021 02:20:47 GMT

Printing a big set of data on console will consume many memory, have a try to output the document to a file and allocate more memory to the job execution.

Regards

Shong

Re: Job losses responsiveness

KKlimczuk1618309742 — Tue, 15 Jun 2021 07:10:49 GMT

Added tFileOutputDelimited - the same result. Also allocated more memory for java, from -Xms256M and -Xms1024M to -Xms512M and -Xms2048M, job losses responsiveness in about 61k documents.

Re: Job losses responsiveness

KKlimczuk1618309742 — Tue, 15 Jun 2021 07:10:54 GMT

Added tFileOutputDelimited - the same result. Also allocated more memory for java, from -Xms256M and -Xms1024M to -Xms512M and -Xms2048M, job losses responsiveness in about 61k documents.

Re: Job losses responsiveness

KKlimczuk1618309742 — Fri, 18 Jun 2021 08:10:16 GMT

Any other help?? To wrap up, no difference if I print or redirect output to file, the same result - freeze, even if I run job only with couchbase node... The job freezes with no response after some number of documents, always around 61k

Re: Job losses responsiveness

Anonymous — Fri, 18 Jun 2021 09:28:06 GMT

@Kacper Klimczuk, I don't have an environment for testing now. Can you please open an issue under Talend DI component project on our bugtracker for our R&D team to investigate it further. Thanks!

Regards

Shong

Re: Job losses responsiveness

KKlimczuk1618309742 — Fri, 18 Jun 2021 12:05:48 GMT

I cant access this site, I guess I dont have permissions. It says that I should contact my Jira administrator but I dont know who to contact

Re: Job losses responsiveness

KKlimczuk1618309742 — Mon, 21 Jun 2021 07:45:58 GMT

I think there could be problem in Java SDK, as I search there is a problem with dcp connection and it is resolved by specifiying few parameters - https://forums.couchbase.com/t/dcp-stops-getting-data/21163. Is there any possibility to specify parameters for tCouchbaseDCPInput??