Reading Flat Large file (2GB) in Big data job (spark)

[Anonymous]

I am trying to process 2gb flat file using spark big data job. 

it takes very long time (3+) to just read the file. 

i have also updated no of executors and executor memory (reluctantly). it doesn't work either. 

any suggestions is appreciated  

 

[manodwhb]

@uganesh , how are you executing your job ? are you executing from Studio?

[Anonymous]

@manodwhb thanks for quick response. 

I am running Job on RE (submit from studio) , hosted on my AWS env and connecting to EMR cluster.  

[manodwhb]

@uganesh, so you were building a job and copying that zip file in the remote engine and executing .sh file?

[Anonymous]

@manodwhb : 

Below is screenshot of my studio. 

Studio ---publish-->  RE --> EMR 

[manodwhb]

@uganesh, all executors are utilizing?