topic Re: Unable to get GlobalMap content from start block in tJavaFlex in Talend Studio

Unable to get GlobalMap content from start block in tJavaFlex

Anonymous — Mon, 23 Nov 2015 17:14:23 GMT

Job flow is shown with screenshot.

tJavaFlex code:

Startcode:
int currentId = (Integer)globalMap.get("out1.ID");
String LogRowIn = (String)globalMap.get("out1.LogRowIn");
for(int i=0; i<((Long)globalMap.get("out1.iterations")); i++) {

MainCode:
row2.ID = currentId;
row2.LogRowOut = LogRowIn;

EndCode:
}

---------------------------------------------
While execution log showing these exception why?

java.lang.NullPointerException
at wurfl_smutility_connector.tflowiteratejob_0_1.tFlowiterateJob.tFileInputRegex_1Process(tFlowiterateJob.java:1391)
at wurfl_smutility_connector.tflowiteratejob_0_1.tFlowiterateJob.tLibraryLoad_1Process(tFlowiterateJob.java:412)
at wurfl_smutility_connector.tflowiteratejob_0_1.tFlowiterateJob.runJobInTOS(tFlowiterateJob.java:1857)
at wurfl_smutility_connector.tflowiteratejob_0_1.tFlowiterateJob.main(tFlowiterateJob.java:1714)

Re: Unable to get GlobalMap content from start block in tJavaFlex

Anonymous — Mon, 23 Nov 2015 23:52:34 GMT

This is because the globalMap variables have not been set at the time the start code is fired. You need to use these variables in the main code.

Re: Unable to get GlobalMap content from start block in tJavaFlex

Anonymous — Tue, 24 Nov 2015 03:48:22 GMT

Thanks Rilhia for you reply, how can i change my tjavaflex code to run the above job.

Re: Unable to get GlobalMap content from start block in tJavaFlex

Anonymous — Tue, 24 Nov 2015 03:57:00 GMT

Is there any chance to read the component column data from start code... Because i want read the all data from start code then will do process from main code using loop

Re: Unable to get GlobalMap content from start block in tJavaFlex

Anonymous — Tue, 24 Nov 2015 09:19:38 GMT

I'm not sure what you are trying to achieve here. Why are you using the tFlowToIterate? Why not connect the tMap to the tJavaFlex? What is the point of iterating over the dataset? I feel there might be a better way of achieving your goal, but would need to know what that goal is to go further.

Re: Unable to get GlobalMap content from start block in tJavaFlex

Anonymous — Thu, 26 Nov 2015 16:13:14 GMT

Thanks to considering my post.

I designed a job using tJavaFlex. In main code these are the steps I followed

1) Read data from input file
2) if readLines=10000, store them into List then process list with externalJar and get outputList .
3) store outputList in 'GlobalMap'
4) clear list.
5) Repeat 2, 3, 4 until EOF and store outputLists to GlobalMap.
6) write records to outputfile

In End code:
I processed last records and stored in GlobalMap
---
Now we have GlobalMap which has list data

But I'm facing issue to process big file(~>1GB) using tJavaFlex because GlobalMap depends on JVM Hap(Ram), I increased Heap(min-4Gb, Max-7Gb).

Can you tell me any component/Process to mitigate my issue?

Re: Unable to get GlobalMap content from start block in tJavaFlex

Anonymous — Thu, 26 Nov 2015 17:26:05 GMT

You don't appear to be using the functionality supplied by the Jar unless you are using this in the tMap. Why is it only processing 10000 rows at a time? You need to break this down into more components. I think you will need a tLoop to do this. I have an example of a tLoop being used in a relatively dynamic way here. It's not what you are doing which will be looping on every 10000 records (without knowing the total number of records), but it does loop on an unknown number of service calls. The looping mechanism should give you an idea how to achieve this.

Re: Unable to get GlobalMap content from start block in tJavaFlex

Anonymous — Thu, 26 Nov 2015 19:56:09 GMT

I like your tutorials, got basic things from your tutorial.

This is the flow I want to do:

tFileInput(Column: Record) ---> tMap(Column: Seq_id,Record)--(iterate)-->tLoop(what can we define here?)--->Get records and process with externaljar---->tFileOutputDelimited

How can I iterate records using tLoop, what components can i use with it?

for example I have 1 billion records,
First I want to iterate loop for 1 million records(iterate loop until seq_id is 1billion) ---->tJavaFlex to Save records in list then list into GlobalMap---> tJavaFlex to process GlobalMap's List--->tJavaFlex to save records output---->tFileOutput

Is it possible? can you give me brief design to do this?

Re: Unable to get GlobalMap content from start block in tJavaFlex

Anonymous — Fri, 27 Nov 2015 01:17:22 GMT

OK, I think I misunderstood before. This is quite complicated since you are not able to query by batch of 10000 records. You have also not mentioned in what format you need to pass the data to your external Jar. You say a List, but I am assuming that you are talking about a List of rows. So how will the rows be held in the List?
What you could do is use a tJavaFlex to iterate over the records and add them to a List in the Main Code. Keep a count in the Main Code and for every 10000 records use an IF condition to send the List to your External Jar. In that IF also empty the List so that a fresh List can be built over the next 10000 records. In the IF also add the returned List to a Collection class (whichever you prefer) and store that Collection class in the GlobalMap variable.

Every time that IF condition is accessed (for every 10000 records), send the List to the External Jar, retrieve the Collection class from the GlobalMap, add the return from the External Jar to the globalMap. Keep doing this until all of the data has been processed and is in a List inside the Collection class inside the GlobalMap.

Then in another subjob, start with a tJavaFlex. In the Start Code retrieve the Collection class from the globalMap and initiate a For Loop based on the size of the Collection class. Then for each iteration of the loop (still in the Start Code) retrieve the List that was returned by the External Jar. Then create another For Loop (still inside the Start Code section) based on the number of rows in that particular List. The rest of the code takes place in the Main Code.
In the Main Code what you are dealing with is the each iteration of the inner loop (the List from the External Jar). Extract the column data and send it to the next component (you will need to create a suitable schema for the tJavaFlex).

Then in the End Code, you will need to close both the loops initiated in the Start Code.

By doing this, you are first (in the first subjob) building a massive collection of data (be careful as memory may be an issue). In the second subjob (starting with the tJavaFlex) you are extracting the rows and returning them to a normal flow.

As I said, this is quite complicated and the sort of thing that I would normally have to demonstrate. I am hoping you are able to extract enough from this brief description to do something similar. But struggling through a problem is far better than being given one I guess 🙂

Re: Unable to get GlobalMap content from start block in tJavaFlex

Anonymous — Fri, 27 Nov 2015 15:46:21 GMT

Thanks for your reply, I found another flow, instead of using RAM why don't I use InMemory with Disk.
I used to store all 2 billion data in GlobalMap and system got hung because of huge JVM heap.
So i got other idea which follows as
tInputFile(Column:Record)---- Split the record into 'n' files ---- Using tJavaFlex read every file, call externalJar(it has method by passby parameter as file) then write back result values into file.
Finally every input splits will interact with tJavaFlex and get n target files in disk.
Then combine all n input records to target file.

What component I can use to split the file into n files?
If possible, What component I can use to combine n files into target files ?
If possible, How to append target data for every file?
Please give me the brief design to proceed.