topic Spark Batch job: Using output of tHiveinput in tSQLRow in Talend Studio

Spark Batch job: Using output of tHiveinput in tSQLRow

Anonymous — Sat, 16 Nov 2024 06:39:50 GMT

I have the below SQL in tHiveInput:

"select max(plan_dk) as MAX_DK from db.table_1"

I would like to use the max_dk in the 'select' statement of tSQLRow. could you please give me an example of doing it.

SQL in the tSQLRow is something like this:

select row17.max_dk+row_number() over(order by null)

from table;

Re: Spark Batch job: Using output of tHiveinput in tSQLRow

vapukov — Wed, 06 Feb 2019 03:01:45 GMT

---

Re: Spark Batch job: Using output of tHiveinput in tSQLRow

manodwhb — Wed, 06 Feb 2019 05:57:39 GMT

@acherukuri ,I suggest yo to create a context in main standard job and call the spark job in main job and enable transmit the while context to that sub spark job.