topic Re: Spark Streaming to hive in Talend Studio

Spark Streaming to hive

Anonymous — Sat, 16 Nov 2024 08:38:06 GMT

Hi All,

I am new to talend. I am trying to build a job which will connect to mapr stream , consume the data and write the data to a hive table.

I am using maprstream input to consume the data which is consuming the data correctly. but when inserting the data into hive using thiveoutput component its not working. I am even not getting any error. Can anybody help here.

Thanks,

Ranjit.

Re: Spark Streaming to hive

Anonymous — Tue, 27 Feb 2018 02:45:50 GMT

Are you Posting to partitioned Hive table (in Append Mode) ? can you please attach the screen shots of the Job and the configurations of the tHive output ?

Re: Spark Streaming to hive

Anonymous — Tue, 27 Feb 2018 14:23:06 GMT

Hi,

No I have not selected partitioned option. And save mode is Append. I am using 6.4 Real time big data platform.

The job is a big data streaming job, with one component tMapRStreamInput Input and the other one is tHiveOutput.

I can see the messages are getting consumed using the tMapRStreamInput, but I can't insert them to hive tables.

Note: Cant attach the screenshot because of organisation security policy.

Thanks,

Ranjit.

Re: Spark Streaming to hive

Anonymous — Thu, 01 Mar 2018 02:06:08 GMT

were the Hive tables created upfront before appending the data ? I have this issue when the Hive tables weren't created upfront and tried to append the data to it.

Re: Spark Streaming to hive

Anonymous — Thu, 01 Mar 2018 02:11:57 GMT

Yes, I have created the hive tables before.The problem is it doesn't give any error as well.

Re: Spark Streaming to hive

Anonymous — Thu, 01 Mar 2018 02:22:05 GMT

can you able to print the data to the console ? if so, i don't see the reason why it won't get inserted into a Hive table.

By the explanation the only thing that can go wrong is the Hive table schema created upfront should match the parsing schema.

Re: Spark Streaming to hive

Anonymous — Thu, 01 Mar 2018 02:25:58 GMT

Yes, I am able to print the data in console. The only thing I am wondering i there are any schema mismatch error, then it should show in the console.

Re: Spark Streaming to hive

Anonymous — Thu, 01 Mar 2018 02:28:12 GMT

schema mismatch will print the data to the console and it doesn't post the data to the hive table, make sure the schema's are defined and i would prefer in using lower case letter's. Please post if that solved the issue after trying that

Re: Spark Streaming to hive

Anonymous — Thu, 01 Mar 2018 02:35:55 GMT

do you mean the tHiveOutput component will itself print the data in case its not able to insert the data. Then its not printing form me as well. I was different component to print the data.

Re: Spark Streaming to hive

Anonymous — Thu, 01 Mar 2018 02:38:55 GMT

Printing the data to the console is through tlog row. sorry for not being clear enough