I have a job with 4 subjobs which each query a different AWS S3 data source via Spectrum and loads them to Redshift in series. This runs fine on my local machine with the JVM arguments set to
-Xms1024M
-Xmx4096M
When i run this on a remote engine via TMC i can see from the execution logs and AWS's cloudwatch console that the first 3 subjobs which deal with the smaller data sets (between 2-8 million rows each) run and complete, but the fourth which is larger (usually around 17 million rows) begins but just hangs. I don't get any more information in the logs even though i've added logging if a component fails or the subjob dies. I can see the query is still going in cloudwatch until i terminate the job via TMC.
I'm guessing it's some sort of memory issue (even though our talend instance has 16Gb memory available and 4Gb seems to be enough on my local machine) but without any feedback from Talend it's difficult to know. I don't even get the out of memory error you would get when running locally.
Has anyone any suggestions how i can get more info on what is going wrong and thus maybe solve the problem?
