This article shows you how to collect information using VisualVM to determine what processes are running when an application is in a Not Responding state.
Installation and startup
-
Download the installer from the VisualVM page (no installation is required).
-
Extract the zip file.
-
Using the .exe file, located in the bin/ subfolder, start VisualVM. It should look similar to this:
Talend Studio appears as Local Application, and Jobs appear as projectName.jobName. Grayed-out applications are no longer running. Double-clicking an application opens a new tab where you can look into the details of that application.
Interpreting the Monitor and Sampler tabs
Monitor
CPU Usage speaks for itself. However, if you see a lot of GC activity, that means the application is swapping and needs more memory. The memory chart on the right indicates this as well. The Heap size is useful in determining how much memory the application needs. The Used heap (marked with blue) is what the application is currently using.
Perform GC
Java relies on a garbage collector (GC) to clean up the memory. It holds objects that could be removed, but only removes them when you're at the Heap size and Used heap limits.
The chart above shows an automatic GC (the big drop), and when GC had been performed (that small drop at the edge), also the CPU usage shows this activity.
How much memory should you assign?
It should be a delicate balance between having the garbage collector constantly running (too little memory) and having a lot of unused heap (too much memory). Having huge drops (90% to 10%) in the memory usage means there's a lot of extra memory allocated to the application. Generally speaking, not having enough memory has obvious signs, such as receiving OutOfMemory errors from the application, compared to having unnecessary extra memory assigned to the application.
Sampler - Memory
Use this chart to determine what objects are occupying the memory.
Per thread allocations using delta values help you understand what threads are using the memory extensively. You can capture, save, and ship snapshots of the heap, but the delta value is not stored, so you'll need multiple snapshots. It is easier to take a screenshot of the Per thread allocations delta values than using Sampler - CPU to take snapshots.
Sampler - CPU
Use this chart to understand why a program is hanging.
The Total Time (CPU) column shows what the CPU time is spent on and helps identify suspicious activity. You need to open each thread that shows interest. The example above displays the main and four Output Stream Monitor activities.
The main thread after an hour of sampling looks like this:
Notice that it is spending time on sleep / loopIdle, most likely because it is waiting for user input.
The Output Stream Monitor is trying to read from a file:
You can use this information to find the root cause. Also, R&D can find what causing the issue and provide a fix.
Collecting data
The profiling method allows you to collect data, then ship it using Profiler Snapshot. Create a profile snapshot by performing the following steps:
- Start sampling.
- Do some interaction.
- Create multiple snapshots (name and save them), for example, break a Studio startup into steps. Each requires some interaction before, so if possible, start the sampling before the user action, and stop after the application is responsive. The snapshots might be:
- Until the Project selection screen comes up
- Project login to Studio screen
- Switching to a different GIT branch
- Opening a Job and running it
-
Finally, export the snapshots as .nps files by right-clicking the snapshot and selecting Save As or using the Save As icon.
Collecting data on Linux Servers (Talend Administration Console, Tomcat, JobServer, Nexus)
You can load Java Flight Recorder (JFR) results to VisualVM, and you can run JFR on Linux servers.
Requirements
Make sure your Java is correct.
- Oracle Java 8 it has this feature available.
- If you're using Zulu Java 8 or OpenJDK Java 8, then make sure you're on the latest version, as the JFR feature was only added in mid-2020.
The following command should be executable:
java -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -version
If Java supports it, you will see:
If Java doesn't support it, it needs to be updated:
Setup
Enable JFR by adding the two parameters mentioned above as JVM parameters.
-XX:+UnlockCommercialFeatures -XX:+FlightRecorder
In the case of Talend Administration Console, this is done in the setenv.sh file; in other cases this can be done where the -Xms/-Xmx parameters are set.
Recording
By default, JFR is in a "sleeping" state and we need to activate it. This can be done using the JCMD utility:
jcmd <PID> JFR.start name=tacRecording filename=/tmp/tacRecording.jfr
The PID can be obtained using ps aux or by running the jcmd command. In the case of Talend Administration Console, you need to look for the Catalina Bootstrap.
You can stop data collection by running:
jcmd <PID> JFR.stop name=tacRecording
If done correctly, you'll see:
Problems
If Talend Administration Console is running with talenduser, then JCMD has to be triggered with talenduser. It can't be executed using sudo / root or it will display the following error:
AttachNotSupportedException: Unable to open socket file: HotSpot VM not loaded
User is correct but JFR is not enabled:
(Start JFR, reproduce the issue, stop JFR, analyze the results with VisualVM.)
If the user is part of multiple groups, use the -g parameter to switch to the right group:
su talenduser -g talendgroup
There should be a file under /tmp called .java_pid<PID> present for the process to be able to attach.
Questions, issues, and solutions
Issue: I can't see my applications.
Solution: Try starting VisualVM with administrator rights.
Issue: I get an error at startup for my Java version.
Solution: If you're able to execute the steps in this article, then it's not an issue.
