Reading a json external table into spark job issue

[smer]

Hello,

I have a Spark Job with a Hive query  that reads an  external table (JSON format), but when i tried to get the colonne INPUT__FILE__NAME i have this error :

hive org.apache.spark.sql.AnalysisException: cannot resolve '`INPUT__FILE__NAME`'

My job

tHiveInput -> tMap -> tHiveOutput

How can i get the file name ?

My config:

Talend Big Data Batch Platform 6.5.1

Cloudera 5.12

Thanks

smer

[Anonymous]

use the row name row1.filename

[smer]

Hello jcruie,

This solution in not working, because the row1.filename is unkown.
my job is
tHiveInput->(row1)->tmap->(row2)->thiveOutput
i use the row1.filename in ten tmap, and in the query hive but not working.

Regards
smer

[Anonymous]

If filename is in tmap you need to use row2.filename

[smer · Accepted solution]

The solution is that "input__file_name" is a virtual hive collumn, i just use the spark's function  input_file_name() and that's it