Slow down while processing 100M+ row file?

[talendtester]

I have a file with over 100 million rows of data.

The job processes around 2,780 files per second when the job starts, but after about 5 million rows the speed starts to slow down and eventually goes down to about 2 rows per second.

The job is:

tFileInputDelimited > tMap > tContextLoad

                             ↓

                         tJava > tFileOutputDelimited

In the tMap component, I have Advanced settings Store on disk Max buffer size: 1,000,000

In the job's Run Tab advanced settings I have: -Xms6256M and -Xmx7024M

The virtual server I am running the job on has 8 processors, 8 sockets and 32GB of RAM

What can I do to keep the job running at 2,780 files per second?

[Anonymous · Accepted solution]

Your routine needs to look something like this (you will need to handle the imports, etc).....

public class GPSConvert {

    public static String ConvertCoords(Double long_, Double lat_){
    	
    	String myResult = "";
    	CoordinateConversion cs = new CoordinateConversion();

    	//GET THE MGRS VALUE:
    	myResult = String.valueOf( cs.latLon2MGRUTM(lat_,long_));
    	return myResult;
    }    
    
}

You can use this in your tMap by simply placing the code below in your the column you want to output this data in....

routines.GPSConvert.ConvertCoords(row1.long, row1.lat)

There may be a bit of tidying up to do, but this will make your job run a lot faster.

[talendtester]

As you suggested, I created a routine to call the MGRS CoordinateConversion routine.

It runs fine for over 2K rows then dies when the input is:    
Latitude:    21.32889
Longitude: -158.12221

Which doesn't make sense because the points are within range and it should return:  04QEJ9102958801

It died with this error:

Exception in component tMap_1 (mgrs)
java.lang.IllegalArgumentException: Legal ranges: latitude [-90,90], longitude [-180,180).
    at routines.CoordinateConversion.validate(CoordinateConversion.java:30)
    at routines.CoordinateConversion.access$1(CoordinateConversion.java:25)
    at routines.CoordinateConversion$LatLon2MGRUTM.convertLatLonToMGRUTM(CoordinateConversion.java:257)
    at routines.CoordinateConversion.latLon2MGRUTM(CoordinateConversion.java:39)
    at routines.MGRS_Convert.CoordinateConversion(MGRS_Convert.java:11)
    at demo.mgrs.mgrs.tFileInputDelimited_1Process(mgrs.java:1287)
    at demo.mgrs.mgrs.runJobInTOS(mgrs.java:2508)
    at demo.mgrs.mgrs.main(mgrs.java:2339)

[Anonymous]

I suspect that you reversed the longitude and latitude given the error message. 

[talendtester]

You are good!

[Anonymous]

I've worked with Long and Lat before. I made the same mistake countless times 🙂