topic Re: How to push data from HDFS to Hive Table in Talend Studio

How to push data from HDFS to Hive Table

Anonymous — Sat, 16 Nov 2024 11:50:23 GMT

Hi All,

I am not able to push data from HDFS to Hive .
Objective :
Create tables and load data into Hive.

Platform :
CDH 4.4 .
TOS for Big Data 5.4.0
Ubuntu OS.

Architecture :
Both CDH and TOS are on same machine(CDH with Single host implementation)
Components being used are :
tHiveConnections
tHiveCreateTable
tHiveLoad
More info :
Screen shot.

Re: How to push data from HDFS to Hive Table

Anonymous — Thu, 14 Nov 2013 09:51:15 GMT

Error Log:
Starting job Test at 06:53 14/11/2013.

connecting to socket on port 4047
connected
: org.apache.hadoop.hive.conf.HiveConf - hive-site.xml not found on CLASSPATH
: org.apache.hadoop.conf.Configuration - mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
: org.apache.hadoop.conf.Configuration - mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
: org.apache.hadoop.conf.Configuration - mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
: org.apache.hadoop.conf.Configuration - mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
: org.apache.hadoop.conf.Configuration - mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces : org.apache.hadoop.conf.Configuration - mapred.reduc

To see the whole post, download it here
OriginalPost.pdf

Re: How to push data from HDFS to Hive Table

Anonymous — Fri, 15 Nov 2013 09:39:24 GMT

You just have a permission issue within HDFS.
Please change that using hadoop fs -chmod and haddop fs -chown

Re: How to push data from HDFS to Hive Table

Anonymous — Fri, 15 Nov 2013 09:42:16 GMT

Hi Remy ,
can you please suggest me on the components i should be using to write data directly from hDFS to Hive table. I tried with tHiveLoad which upload data from file and not from HDFS file . Can you please guide me . Currently i am pushing or writing data to a file on linux box and then fetch that to hive tables using tHiveload.

Re: How to push data from HDFS to Hive Table

Anonymous — Fri, 15 Nov 2013 10:08:56 GMT

Hi,
If your data is already on HDFS, then I think you can use the tHDFSCopy to move the data from the current location to the HDFS location of your Hive table.
Does it make sense?

Re: How to push data from HDFS to Hive Table

Anonymous — Fri, 15 Nov 2013 10:11:13 GMT

Hi Remy,
Makes sense . Thanks a lot. Can you do me another favor . Can you suggest any better BI engine which would fit better with Talend Open Studio for Big Data. Currently , i am trying with SpagoBI and find it pretty confusing and complex. Thanks.

Re: How to push data from HDFS to Hive Table

Anonymous — Tue, 26 Aug 2014 11:40:46 GMT

Hi
I think problem in your tHiveLoad component.Your local has been checked.Checked it out and give the hdfs file path,it will load data to your Hive Table directly.