We have a standard job calling 60 Spark jobs using tRunJob component. we are not really able to launch 60 threads on EMR spark cluster. when we launch jobs we see that all 60 threads create 60 processes but shortly process count drops to 17 and only those 17 threads(tRunJObs) are passed to Spark EMR from job server. does anyone able to run 50+ Big data jobs from job server successfully? if yes kindly share some details.
