Folks, The input to my talend job is two MQs, each queue gets loaded with a position delimited file one Queue 1: Business Events (one event per line) Queue 2: Event freeform data. Which represents potentially 0-* rows per event. Tied together by the EVENT ID which is present at a position point in both files. Now, obviously, to process an event in its entirety, I need to load the entire EVENT freeform queue (queue 2) before matching that data up with the EVENT data from Queue 1. So, my processing is as follows: tMap matches event to event freeform data rows. Feeding into the tMap is the main input of the event queue output after being positionally parsed then the other input is a lookup with lookup model set to "Load Once", Match Model "All Matches". Purely because this data is unfortunately not tied in any way to the Message ID in the MQ queue, I can't easily match the two sets of data up without loading in all of the freeform data first - I have no control over the nature of the inbound event data structure. I am open to suggestions here. But onto my main question. The loading of the MQ freeform data through the parsing it out based on the positional delimited rules (all done in memory) is taking a very long time and I'm not sure if it is to be expected OR I can tweak something to improve my flowthrough rates. Currently for 1.8 Million MQ freeform rows, it takes my job about 38 minutes to load every message from the queue (using tMomInput). From there, the positional parsing (using tFilePositional taking input as a datastream from the MQ fetch subjob) takes approximately 2 hours to run before the tMap has it's lookup data to perform the rest of its function. All of this is running on a Websphere application server on my development machine (core i5 laptop with 8Gb RAM) with my JVM under Websphere configured for 1Gb of heap. How does that sound? Is this a case of normal performance speeds based on the environment or something else?
xdshi,
Thanks for getting back to me but I've decided to move on from this issue. My real area of concern was the fact that my freeform data had to fully load into memory prior to being used as a lookup within my tmap. From what I can tell, there is simply no way around that. The tmap will need to load all of the data first to begin processing.
thanks,
