I am facing issue with Talend jobs scheduled on Production. The jobs are taking too much time than usual to complete. There is no change in data volume or any modifications made on the job. I started experiencing this issue recently and has occurred 3 times now. I have to restart the RE instance to get that fixed. Once the RE service is restarted, the jobs used to run normally and gets completed in the expected time. Once more observation, I have noticed is that when the jobs start taking more time to run, the logs just show minimal information, but wont show the execution details which we used to see otherwise. When we restart the RE instance, it used to work normally. I have checked for the memory usage, CPU utilization and all but it seems normal.
Please help me in understanding why this issue is coming frequently.
The issue where Talend jobs slow down, show minimal logs, and require a Remote Engine (RE) restart suggests a resource bottleneck (memory/CPU) or a deadlock within the Java Virtual Machine (JVM) running the engine, potentially caused by excessive log data or faulty garbage collection. The temporary fix via restart strongly points to a buildup that is cleared upon reboot.
Here are the key troubleshooting steps based on your scenario:
Disable Stats and Logs: The most likely cause for low-level freezing and missing log details is the overhead from excessive logging. Disable "Use statistics," "Use logs," and "Use Volumetrics" in the Job setting's Stats & Logs section.
Check Remote Engine Memory: Monitor the RE for OutOfMemory errors or severe JVM garbage collection pauses. If memory is maxed out, increase the JVM heap size allocated to the Remote Engine.
Clean Temp/Work Directories: On the machine hosting the Remote Engine, clear the apache-tomcat/work and apache-tomcat/temp folders to remove stale session or cached data.
Check for Network Latency: Intense network latency between the Talend Cloud and the Remote Engine can cause tasks to hang in "Running" status, as result logs cannot be sent back effectively.
Database Locks: Check for deadlocks or long-running queries on the target database, which can cause the job to wait, causing a slowdown.
If the issue persists, try reviewing the thread dump during a "slow" period to see which process is blocking the job.
What I have noticed is that all the jobs takes around 2 minutes to start itself. That's the job will be in the running status, but won't be doing any processing. This happens not just for single job but for all the jobs.
But once, I restart the RE service, there won't be any 2 mins of waiting time and the processing starts instantly.
I have checked for the Memory usage and is just in the range 1-2 GB out of 15GB available and the CPU usage is below 50% all the time.
Can u let me know from where I can get the thread dump as well.
