Usage of Thiveinput along with TSqlRow

Anonymous — Sat, 16 Nov 2024 08:26:12 GMT

I am new to using Big Data Batch jobs but i have previously worked on standard jobs for developing big data solutions. I am currently trying to build a spark job which uses a Thiveinput (Hive Table) as source and write to Thiveoutput (Hive Table). I am getting the below mentioned issues :

[WARN ]: org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
org.apache.spark.sql.catalyst.parser.ParseException:
no viable alternative at input '<EOF>'(line 1, pos 4)

== SQL ==
USE
----^^^

at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:197)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:99)
at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:46)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:53)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:582)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:682)
at kai.dummy_0_1.Dummy.tHiveInput_1Process(Dummy.java:1034)
at kai.dummy_0_1.Dummy.run(Dummy.java:1432)
at kai.dummy_0_1.Dummy.runJobInTOS(Dummy.java:1304)
at kai.dummy_0_1.Dummy.main(Dummy.java:1194)

I have developed a similar job with Parquet as input and written the data into another Parquet file using tsqlRow but i am unable to replicate the same with the hive components.

Adding a screenshot of the jobs below ( Just the Thiveinput & Tlogrow )

Thanks for the help in advance.

Aditya

Re: Usage of Thiveinput along with TSqlRow

Anonymous — Fri, 13 Apr 2018 08:32:28 GMT

Hello,

So far, tHiveRow and tHiveCreateTable components will be available in Standard Job not bigdata batch job.

If you use Parquet components, you can already partition with Spark and easily combined with Hive DI createTable (declaring it as an external table).

Best regards

Sabrina

topic Usage of Thiveinput along with TSqlRow in Talend Studio

Usage of Thiveinput along with TSqlRow

Re: Usage of Thiveinput along with TSqlRow