topic Data Inetgration in TOS hangs alot? in Talend Studio

Data Inetgration in TOS hangs alot?

Anonymous — Thu, 20 Apr 2017 19:32:42 GMT

Hi,

i am using Windows 7, 64 bit machine. I have increased the JRM paramters of memory space to **2046M**. Still after few hours, the system hangs alot.

Could anyone explain what is the reason?

(I am doing no database work, only reading from Excel files and string in Excel files.)

Any good suggestions, much appreciated! (Below is screenshot how clicking an icon make the whole tool stuck for minuetes 15 or 20)

Re: Data Inetgration in TOS hangs alot?

vapukov — Thu, 20 Apr 2017 19:47:01 GMT

There are many reasons, but one of most resent problems with studio - it is error log (tLogRow and other output)

like, if looking for Your picture:

possible to think - this is for each row in Your flow. If it true - it is exactly source of problem

Why - I can not answer, but can surely confirm - big number of output rows hang Talend always

hang - in best case, in worst it will be crashed

Re: Data Inetgration in TOS hangs alot?

cterenzi — Thu, 20 Apr 2017 19:53:17 GMT

I also have problems when I have a lot of data in the console. Clear on run should be enabled unless you need to see the output of multiple runs. Manually clearing the console can help if you've just dumped a lot of data there.

Re: Data Inetgration in TOS hangs alot?

vapukov — Thu, 20 Apr 2017 19:58:57 GMT

there 2 cases:

1) if it tLogRow and number of rows more than "few" - better to change tLogRow by tFileOutputDelimited - work fast and not eat memory

2) but if it result of errors - for example for each row return warning about wrong format or similar ... this is possible to fix only by resolving source of problem

Re: Data Inetgration in TOS hangs alot?

Anonymous — Thu, 20 Apr 2017 20:45:33 GMT

how to enable that?

Re: Data Inetgration in TOS hangs alot?

vapukov — Thu, 20 Apr 2017 20:46:35 GMT

what?

Re: Data Inetgration in TOS hangs alot?

Anonymous — Thu, 20 Apr 2017 20:48:12 GMT

@cterenzi Could you tell how to enable "Clear on run should be enabled"

Re: Data Inetgration in TOS hangs alot?

vapukov — Thu, 20 Apr 2017 20:50:01 GMT

it enabled by default, but it not help if I right about number of rows on the picture - it is a killer

Re: Data Inetgration in TOS hangs alot?

Anonymous — Thu, 20 Apr 2017 20:51:33 GMT

Yes, the data size is like 5 excel files with 26K rows in each file.

Re: Data Inetgration in TOS hangs alot?

vapukov — Thu, 20 Apr 2017 20:52:28 GMT

redirect this output into csv file (or disable) and all will be fine

Re: Data Inetgration in TOS hangs alot?

cterenzi — Thu, 20 Apr 2017 20:52:49 GMT

Run Tab, Advanced Settings.

Re: Data Inetgration in TOS hangs alot?

vharcq — Fri, 21 Apr 2017 05:35:13 GMT

Classic problem indeed, the "Console" in "Run job" view is a full memory component

WIth too much data it just died (and kill the whole studio)

You need to tick the "Line limit" and choose a max number of line during your test run

If you really need more data when you debug, you could try a new Log4j FILE appender to a file in "File / Edit Project Properties / Log4j"

Re: Data Inetgration in TOS hangs alot?

Anonymous — Sun, 10 Dec 2017 11:58:57 GMT

My updated Log4j settings looks like below.

Just add a new file appender (i.e. with 'class' of 'org.apache.log4j.FileAppender'), add reference to it at bottom in 'root' section, and comment the 'CONSOLE' appender so it stops crashing Studio).

if on Windows, UnxUtils has a 'tail' command you can use: http://unxutils.sourceforge.net/

=======================================================

<appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern"
value="[%-5p]: %c - %m%n" />
</layout>
</appender>

<appender name="FILE" class="org.apache.log4j.FileAppender">
<param name="file" value="c:/log/studiojob.log" />
<param name="append" value="false" />
<param name="threshold" value="trace" />
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{ISO8601} %-5p [%c{1}] - %m%n" />
</layout>
</appender>

<root>
<priority value ="warn"/>
<appender-ref ref="FILE"/>


</root>

=======================================================