topic Re: [resolved] Sending message to tFlowMeterCatcher in Talend Studio

[resolved] Sending message to tFlowMeterCatcher

blh — Fri, 11 Dec 2015 15:52:05 GMT

Hello,
I would like to count data inserted into my table. The usual way is to insert tFlowMeter component on the row link to perform this. But as you can see on the screenshot below, I use 'bulk' components to load the data into the table for performance purpose. So there is no row link outgoing from the tOracleBulkExec component to connect to a tFlowMeter component.
However, the tOracleBulkExec component stores these information into some global variables, and I would like to send the values of these variables to the tFlowMeterCatcher.
I tried to connect a tJava to the tOracleBulkExec with the following java code but it seems this does not work (no trace on the console):

talendMeter_METTER.addMessage("DATA_INSERTED", ((Integer)globalMap.get("tOracleBulkExec_1_NB_LINE_INSERTED")), "null", "", "");

Does someone know a workaround or another way to send a message to the tFlowMeterCatcher ?
Thanks in advance for your help.
Bruno

Re: [resolved] Sending message to tFlowMeterCatcher

Anonymous — Mon, 14 Dec 2015 04:14:23 GMT

Hi
You should use a tFlowMeterCatcher in the job and change the value of count column with the global variable ((Integer)globalMap.get("tOracleBulkExec_1_NB_LINE_INSERTED")). eg:
...tOracleBulkExec--oncomponentok--tFixedFlowInput--main--tFlowMeter--tJavaRow
tFlowMeterCatcher--main--tMap--main--tLogRow

on tMap: change the value of count column with the global variable.

Re: [resolved] Sending message to tFlowMeterCatcher

blh — Mon, 14 Dec 2015 10:28:13 GMT

Hi Shong,
First, thanks a lot for your reply.
I just tried your workaround, and despite it works pretty well, it is not as generic as I could expect because I need to specify explicitly the variable in the tMap. After a test run (check at the trace below), I identify a few problems with this approach:
1) Things become quite complicated if you need to log multiple variables using this way in the job such as NB_LINE_INSERTED and NB_LINE_BAD for example.
2) You need to filter the input of the tMap to avoid null pointer exception if you use another tFlowMeter before the tOracleBulkExec which is the case in my example (in the previous subjob).

3) I receive two logs for my variable, and I do not understand why. Maybe I made a mistake in the tMap configuration (I attached a screenshot of the mapping).

[statistics] connected
2015-12-14 10:28:38|QuXGHA|QuXGHA|QuXGHA|6372|OPCO|Opco_03b_LoadIndex|_OqJUQJ9dEeWv45XULJuvcg|1.0|Developpement||begin||
2015-12-14 10:28:38|QuXGHA|QuXGHA|QuXGHA|OPCO|Opco_03b_LoadIndex|Developpement|1|tWarn|tWarn_2|Started...|10
2015-12-14 10:28:46|QuXGHA|QuXGHA|QuXGHA|6372|OPCO|Opco_03b_LoadIndex|_OqJUQJ9dEeWv45XULJuvcg|1.0|Developpement|tFlowMeter_3|INDEX_FILE_READ|1638||
2015-12-14 10:28:46|QuXGHA|QuXGHA|QuXGHA|6372|OPCO|Opco_03b_LoadIndex|_OqJUQJ9dEeWv45XULJuvcg|1.0|Developpement|tOracleBulkExec_1|begin||

SQL*Loader: Release 10.2.0.3.0 - Production on Lun. Dc. 14 10:28:46 2015

Copyright (c) 1982, 2005, Oracle.  All rights reserved.
2015-12-14 10:29:30|QuXGHA|QuXGHA|QuXGHA|6372|OPCO|Opco_03b_LoadIndex|_OqJUQJ9dEeWv45XULJuvcg|1.0|Developpement|tOracleBulkExec_1|end|success|43217
2015-12-14 10:29:30|QuXGHA|QuXGHA|QuXGHA|6372|OPCO|Opco_03b_LoadIndex|_OqJUQJ9dEeWv45XULJuvcg|1.0|Developpement|tFlowMeter_5|INDEX_DATA_INSERTED|236784||
2015-12-14 10:29:30|QuXGHA|QuXGHA|QuXGHA|6372|OPCO|Opco_03b_LoadIndex|_OqJUQJ9dEeWv45XULJuvcg|1.0|Developpement|tFlowMeter_5|INDEX_DATA_INSERTED|1||
2015-12-14 10:29:30|QuXGHA|QuXGHA|QuXGHA|6372|OPCO|Opco_03b_LoadIndex|_OqJUQJ9dEeWv45XULJuvcg|1.0|Developpement||end|success|51955
[statistics] disconnected

By the way, even if this is not the purpose of this post (neither the right place), it could be more interesting if the tFlowMeter could accept both row and trigger input links, and in the case of the second one, its component view could thereby provide an additional parameter to let the job designer decide how to set the count value.

Thanks again for your help.
Regards.
Bruno

Re: [resolved] Sending message to tFlowMeterCatcher

blh — Mon, 14 Dec 2015 13:22:07 GMT

Hi again,

Based on the solution you proposed earlier, I just found a way to send a log message for multiple variables and to have a more generic process around the tFlowMeterCatcher.

Here is the approach I implemented.
The part which send the message:
[tOracleBulkExec] -- OnComponentOk --> [tSetGlobalVar] -- row1 --> [tFlowMeter_1] -- row2 --> [tFlowMeter_2]
The tSetGlobalVar stores the values of the two variables of the tOracleBulkExec (NB_LINE_INSERTED and NB_LINE_BAD), as follows:
"DATA_INSERTED" = ((Integer)globalMap.get("tOracleBulkExec_1_NB_LINE_INSERTED"))
Each tFlowMeter send the message to the catcher and uses the same string in its label parameter that the one used for the key of the globalMap to store the value.

The part which receive the message:
[tFlowMeterCatcher] -- row --> [tUniqRow] -- row (Uniques) --> [tJavaRow] -- row --> [tLogRow]
The tUniqRow is an attempt to keep a unique message based on the "origin" field (see the explanation below).
The tJavaRow checks whether the label is contained in the globalMap before replacing the count field as the following code explains:

output_row.count = ((Boolean) globalMap.containsKey(input_row.label)) ? ((Integer) globalMap.get(input_row.label)) : input_row.count;

However, it still remains a problem as the execution trace shows. Each log message are sent twice and I do not understand why. I also tried to insert a tUniqRow between the tFlowMeterCatcher and the tLogRow to keep only unique messages based on the "origin" field as a key. But this does not resolve the problem.
I suspect that the tFlowMeterCatcher sends FlowMeter messages despite everything, while there is an output sending the messages to a tLogRow.

[statistics] connected
2015-12-14 13:21:41|DF5sFZ|DF5sFZ|DF5sFZ|6260|OPC|Opc_03b_LoadInd|_OqJUQJ9dEeWv45XULJuvcg|1.0|Developpement||begin||
2015-12-14 13:21:41|DF5sFZ|DF5sFZ|DF5sFZ|OPC|Opc_03b_LoadInd|Developpement|1|tWarn|tWarn_2|Started...|10
2015-12-14 13:21:50|DF5sFZ|DF5sFZ|DF5sFZ|6260|OPC|Opc_03b_LoadInd|_OqJUQJ9dEeWv45XULJuvcg|1.0|Developpement|tFlowMeter_3|FILE_READ|1638||
2015-12-14 13:21:50|DF5sFZ|DF5sFZ|DF5sFZ|6260|OPC|Opc_03b_LoadInd|_OqJUQJ9dEeWv45XULJuvcg|1.0|Developpement|tFlowMeter_3|FILE_READ|1638||
2015-12-14 13:21:50|DF5sFZ|DF5sFZ|DF5sFZ|6260|OPC|Opc_03b_LoadInd|_OqJUQJ9dEeWv45XULJuvcg|1.0|Developpement|tOracleBulkExec_1|begin||

SQL*Loader: Release 10.2.0.3.0 - Production on Lun. Dc. 14 13:21:50 2015

Copyright (c) 1982, 2005, Oracle.  All rights reserved.
2015-12-14 13:22:35|DF5sFZ|DF5sFZ|DF5sFZ|6260|OPC|Opc_03b_LoadInd|_OqJUQJ9dEeWv45XULJuvcg|1.0|Developpement|tOracleBulkExec_1|end|success|44980
2015-12-14 13:22:35|DF5sFZ|DF5sFZ|DF5sFZ|6260|OPC|Opc_03b_LoadInd|_OqJUQJ9dEeWv45XULJuvcg|1.0|Developpement|tFlowMeter_5|DATA_INSERTED|236784||
2015-12-14 13:22:35|DF5sFZ|DF5sFZ|DF5sFZ|6260|OPC|Opc_03b_LoadInd|_OqJUQJ9dEeWv45XULJuvcg|1.0|Developpement|tFlowMeter_14|DATA_BAD|0||
2015-12-14 13:22:35|DF5sFZ|DF5sFZ|DF5sFZ|6260|OPC|Opc_03b_LoadInd|_OqJUQJ9dEeWv45XULJuvcg|1.0|Developpement|tFlowMeter_5|DATA_INSERTED|1||
2015-12-14 13:22:35|DF5sFZ|DF5sFZ|DF5sFZ|6260|OPC|Opc_03b_LoadInd|_OqJUQJ9dEeWv45XULJuvcg|1.0|Developpement|tFlowMeter_14|DATA_BAD|1||
2015-12-14 13:22:35|DF5sFZ|DF5sFZ|DF5sFZ|6260|OPC|Opc_03b_LoadInd|_OqJUQJ9dEeWv45XULJuvcg|1.0|Developpement||end|success|53935
[statistics] disconnected

Any help would be appreciated.
Regards.
Bruno

Re: [resolved] Sending message to tFlowMeterCatcher

blh — Mon, 14 Dec 2015 17:10:27 GMT

End of the story...
I have an explanation why the log messages appears twice in my console and my log file. The "Logs & stats" parameters of my project were still enabled.
By the way, I am still interested to have your point of view about my workaround, even if I am still convinced that the best way to deal with this use case should be to extend the capabilities of the tFlowMeter component, especially to support iterate, row and trigger links as input, and provide a parameter in the component view to let the job designer setting the value according to its needs.
Best regards.
Bruno

Re: [resolved] Sending message to tFlowMeterCatcher

Anonymous — Tue, 15 Dec 2015 08:33:33 GMT

[size=2][font=Verdana, Helvetica, Arial, sans-serif] [tSetGlobalVar] -- row1 --> [tFlowMeter_1] -- row2 --> [tFlowMeter_2][/font][/size]

I don't understand why you use two tFlowMeter components here? If you have several tFlowMeter components in the job, you need to filter the messages based on the origin filed on tMap, and set the value of count field with the global variable.

Re: [resolved] Sending message to tFlowMeterCatcher

blh — Tue, 15 Dec 2015 09:22:14 GMT

Hi Shong,
Thanks for your reply.
As I mentioned in a previous post, I need to log to separate variables setted by the tOracleBulkExec component (NB_LINE_INSERTED and NB_LINE_BAD in my case). This is why I use two tFlowMeter components.
So, in that case, I also need to distinguish after the tFlowMeterCatcher what variable I need to retrieve to replace the count value depending on the origin of the message.
Maybe there is a more efficient (or Talend) way to do that, but this one works well, and with this generic mechanism I can now log with tFlowMeter as many variables as I wish wich are not based on row flow.
So thank you again to put me on a way to find a solution to solve this problem.
Regards.
Bruno