topic Re: tMap order not correct? in Talend Studio

tMap order not correct?

Anonymous — Tue, 11 Oct 2016 10:05:45 GMT

Hello,
I am curretly facing a problem with the tMap row's order.
As you can see on the screenshot I have several tMap connected to each others. Some tmap have Lookup and 2 main rows with order 1 and 2.

When I execute the job, datas seem to ignore the row with "order 1". Datas are supposed to be checked in order to update a variable (thanks to tJavaRow) to determine if there is a problem in datas consistency or not. If there is a problem, we update a context variable which is used in tMap "filtre_nb_rejects==0" to avoid a "NumberFormatException" when we cast numbers.
The fact the rows with order:1 are ignored makes the entire records circulate without being checked, causing an error when datas are not correct.

I would really be thankful if anyone could help me.

Re: tMap order not correct?

Anonymous — Wed, 12 Oct 2016 06:29:42 GMT

I am just wondering if tJavaRow is the villian here. tJavaRow executes for every input row. Can you try to remove tJavaRow once and see if Main Order 1 still remains in "Starting" status.

Re: tMap order not correct?

Anonymous — Wed, 12 Oct 2016 09:12:13 GMT

This is not going to work and if it does it will not be reliable. You are treating a "row" connector like an "iterate" connector and expecting data rows to be passed through the process in such a way that context variables can be assigned and used per data row. You need to break this down into several subjobs or change the way you handle the error check. I would carry out the initial check on your data. Add the check result to a column on the data row. Then use that column to trigger other logic. If you want to use a context variable for this, you will need to break it down into several subjobs and use the "iterate" connector to ensure timing.

Re: tMap order not correct?

Anonymous — Wed, 12 Oct 2016 13:58:15 GMT

Hello

I see order 1 is a runIf connector in this job, the component or subjob linked with runIf only runs when the execution of previous component finishes. I don't understand the problem here, can you upload a full screenshot of the job and explain what are your expected result?

Regards
Shong

Re: tMap order not correct?

Anonymous — Wed, 12 Oct 2016 14:02:14 GMT

I agree with Rhall_2.0 here. You should probably try the approach that he has suggested

Re: tMap order not correct?

Anonymous — Wed, 12 Oct 2016 14:30:04 GMT

Hi,

thank you all for your answers.

We know that this job could have been developed much better and is not very "optimized". But these developments have been made by a person who was new to Talend and actually is no more in the company right now.

shong: the tMap connectors are not "runIf" connectors, these are main connectors. I can't really screenshot the entire job because it is huge and wouldn't really help for the problem I am facing.

rhall_2.0: I thought the execution of "main order 2" would be waiting for all rows to circulate in the "main order 1". I understand the approach you suggest but the entire project has been developed by another person and I am new to this project and now have to update/improve some part of the jobs. I can't really "repair" this part of the job because it would take too much time. It is working well in production environment at the moment. I just noticed that "problem" when I was running several tests on a new feature I added, I can't even tell you if this behavior is new, but I don't think it is.

What I just can't understand is that the "main order 1" is waiting for the ENTIRE rows to circulate in the "main order 2" connector. That's not even "parallel" or something, the "main order 1" connector remains "Starting" while the "main order 2" is executing.

(sorry for my english which sometimes is approximative)

Re: tMap order not correct?

Anonymous — Wed, 12 Oct 2016 15:30:47 GMT

You will need to give us more information about the code in the tJavaRows before we can help any further I am afraid. However, your understanding of the ordering of the tMap out links is wrong. In a system where there are no hold-ups in the flow, the ordering would be order 1, then order 2 per row. As an example, I have quickly knocked up a test job to show this. It has a tRowGenerator, a tMap and two tLogRows. The tRowGenerator simply produces a sequence of consecutive numbers. The tMap adds either "AAA" or "BBB" to the number and sends one to one output and the other to the other. The order of data is below...
1AAA
1BBB
2AAA
2BBB
3AAA
3BBB
4AAA
4BBB
5AAA
5BBB

The "AAA" data came from order 1 and "BBB" came from order 2.

Re: tMap order not correct?

Anonymous — Thu, 13 Oct 2016 05:18:12 GMT

@predz - Are you sure it is working fine in your production box? Is it possible that this scenario never occured in your production box box because of which you didn't encounter this issue.
Using tJavaRow is the problem here. Some rework will be required here to make this job more robust else this job may fail with another scenario.