topic Re: Reading a json external table into spark job issue in Talend Studio

Reading a json external table into spark job issue

smer — Sat, 16 Nov 2024 08:00:48 GMT

Hello,

I have a Spark Job with a Hive query that reads an external table (JSON format), but when i tried to get the colonne INPUT__FILE__NAME i have this error :

hive org.apache.spark.sql.AnalysisException: cannot resolve '`INPUT__FILE__NAME`'

My job

tHiveInput -> tMap -> tHiveOutput

How can i get the file name ?

My config:

Talend Big Data Batch Platform 6.5.1

Cloudera 5.12

Thanks

smer

Re: Reading a json external table into spark job issue

Anonymous — Tue, 03 Jul 2018 04:54:03 GMT

use the row name row1.filename

Re: Reading a json external table into spark job issue

smer — Tue, 03 Jul 2018 11:22:58 GMT

Hello jcruie,

This solution in not working, because the row1.filename is unkown.
my job is
tHiveInput->(row1)->tmap->(row2)->thiveOutput
i use the row1.filename in ten tmap, and in the query hive but not working.

Regards
smer

Re: Reading a json external table into spark job issue

Anonymous — Tue, 03 Jul 2018 12:28:36 GMT

If filename is in tmap you need to use row2.filename

Re: Reading a json external table into spark job issue

smer — Mon, 11 Feb 2019 15:30:46 GMT

The solution is that "input__file_name" is a virtual hive collumn, i just use the spark's function input_file_name() and that's it