Job size error exceeded

[stephbzr]

Hello,

I have the following error: Exception in thread "main" java.lang.Error: Unresolved compilation problem: The code of method tFileInputDelimited_1Process(Map<String,Object>) is exceeding the 65535 bytes limit

I know this is due to my job being too large. The problem is that I don't know how I should split it. 

Here is my job and its description:

 

I am processing files with several thousand rows where the data integration is done in 3 tables, one of which contains more than 100 columns. There are two main parts : 

• a first tJavaFlex (about 800 rows) accompanied by a tMap that allows me to extract all the data from the file and to perform some transformations. 
• a second tJavaFlex (currently 500 rows, but this will evolve over time) accompanied by a tMap to process the data and define other columns from them. 

Knowing that it is when I activate the second tDBOutput (integration in the largest table) that causes the error. So yes I have already sorted my job, but I can't remove any more components, they are all important, even all the tDBInputs because with the data in the file, I need to search the database to create other columns to integrate into the database. 

For me I should separate the two big parts into two subjobs, but how to transfer more than 100 columns from the tMap into a subjob?

Or the 3 tDBOutputs for integration send them to a subjob, but the problem is the same.

In any case, I know that what is the most voluminous are the 2 tJavaFlex. Only I don't know how to divide this job.

[Anonymous]

Hi

Have a try to move the part 3 (or other parts like this) to next subjob, eg:

...tMap(FAM_CODE)--out1--tHashOutput if the data set is small (or tFileOutputDelimited: write the data to temporary file)

|onsubjobok

tHashInput ( or tFileInputDelimited)--main--tJavaFlex--main--tReplicate...other components....

Regards

Shong

[stephbzr]

Hello @Shicong Hong​ 

Thanks for the illustration with the components, but I'm not sure I get the idea. If I reproduce this illustration in the same job, won't it make the job heavier? Because I'm not removing any components and adding two with the "tHash" or "tFile". So surely the Java code is getting heavier. I was thinking that maybe I should create another job and run it in this job with "tRunJob", but if I can do everything in the same job and avoid this, even better...can you confirm ?