Multiple lookups not supported in tmap component of spark framework talend
When i tried to use more than one lookups with different joining condition in tmap of spark talend getting the following warning,
"All loopup tables must have same expression , this expressin does not exist in other lookup".
I need to use around 10 lookup table join in tmap, if use one tmap for one join my job will be more complicated and also has performance issue.
Please suggest a work around.
Error message :
org.talend.bigdata.dataflow.SpecException: Invalid input accessor: row2.null at org.talend.bigdata.dataflow.hmap.HMapSpec$JoinDef.deserialize(HMapSpec.java:1206) at org.talend.bigdata.dataflow.hmap.HMapSpec$JoinDef.access$1800(HMapSpec.java:1122) at org.talend.bigdata.dataflow.hmap.HMapSpec.joinKey(HMapSpec.java:563) at org.talend.bigdata.dataflow.hmap.HMapSpecBuilder.joinKey(HMapSpecBuilder.java:171) at star_analytics_talend_1.ne_switch_tran_stg_1_0_1.NE_SWITCH_TRAN_STG_1.tFileInputDelimited_1_HDFSInputFormatProcess(NE_SWITCH_TRAN_STG_1.java:3092) at star_analytics_talend_1.ne_switch_tran_stg_1_0_1.NE_SWITCH_TRAN_STG_1.run(NE_SWITCH_TRAN_STG_1.java:3611) at star_analytics_talend_1.ne_switch_tran_stg_1_0_1.NE_SWITCH_TRAN_STG_1.runJobInTOS(NE_SWITCH_TRAN_STG_1.java:3532) at star_analytics_talend_1.ne_switch_tran_stg_1_0_1.NE_SWITCH_TRAN_STG_1.main(NE_SWITCH_TRAN_STG_1.java:3418)
You can use SQL-Row component which is available for spark big data jobs.
The SQL-Row component uses the label of its input link to name the registered table that stores the Data sets from the same input link. For example, if a input link is labeled to Row 1 , this Row 1 is automatically the name of the table in which you can perform queries.
