topic Re: unable to connect hadoop using Talend Bigdata in Talend Studio

unable to connect hadoop using Talend Bigdata

Anonymous — Sat, 16 Nov 2024 07:25:23 GMT

Hi,

I have installed Hadoop in windows machine and tried to connect from Talend big data open studio, but I am unable to connect.

I am getting 'Compilation' error as follows.

Is it related to any jar files to be downloaded??

I am unable to download some optional packages, it is failing and tried multiple times.

Re: unable to connect hadoop using Talend Bigdata

Anonymous — Wed, 24 Oct 2018 07:56:24 GMT

Hi,

It seems some of the JARs are missing in your case. Could you please specify the version you are using and the distribution you are trying to connect?

When you are adding the component for the first time, it will list all the required jars at the top of the component tab for that component to execute correctly. Could you please cross check which one is missing there and try to install it again?

Sometimes, if you are having any restrictions to access internet, you will not be able to download the Jars seamlessly. Could you please check above items and let us know your findings with screen shots?

Warm Regards,

Nikhil Thampi

Re: unable to connect hadoop using Talend Bigdata

Anonymous — Wed, 24 Oct 2018 10:20:26 GMT

Hi Nikhil,

I am using Talend Open Studio for Big Data 7.0.1 version and new to Hadoop, downloaded and configured in windows machine. Hadoop version is 2.7.7 .

Version:

2.7.7, rc1aad84bd27cd79c3d1a7dd53204a8c3ee1ed3ac

NB: I am unable to download some optional packages and tried multiple times all the time download is failing.

Is there any other way to download those packages. I tried using Help->Install Additional Packages.

Re: unable to connect hadoop using Talend Bigdata

Anonymous — Wed, 24 Oct 2018 10:45:16 GMT

@xdshi might be having more idea in this area. Adding her to the loop.

Re: unable to connect hadoop using Talend Bigdata

Anonymous — Wed, 24 Oct 2018 11:04:09 GMT

Hello,

Are you getting some certificate issue when download third-party libraries for your talend open studio for bigdata?

What's JDK version are you using? Oracle JDK 1.8? Please ensure JDK bin is included in your PATH environment variable and is being reached.

More information will be highly appreciated.

Best regards

Sabrina

Re: unable to connect hadoop using Talend Bigdata

Anonymous — Wed, 24 Oct 2018 15:08:48 GMT

Hi @xdshi,

First of all thank you for your valuable comments and i have listed my comments below,

1. I am using Oracle JDK 1.8.0 and configured 'bin' path in system environment variables.

2. I have downloaded many packages (both required and optional) by Help-> Install Additional Packages tab.

3. But still some optional packages are there which is unable to download (refer the attached logs and below snapshots).

4. My concern is about the compilation error (mentioned in my topic) which I am getting in tHDFSConnection component and I am not able to see any required library which needs to be download from the same component.

Please share your views on this.

NB: Hope there is no proxy configuration needed, cause I have already downloaded libraries without changing the same.

seems this error blocking other package downloads

It seems this error (above) blocking all other library download....
list of packages

--------------------------------------------------------------------------

Regards,

Vibin CT.

error_log_1.txt
error_log_2.txt

Re: unable to connect hadoop using Talend Bigdata

Anonymous — Thu, 25 Oct 2018 04:29:40 GMT

Hello,

Do you have a proxy / firewall? Have you tried to restart your Talend Studio for certain additional packages to take effect?

Best regards

Sabrina

Re: unable to connect hadoop using Talend Bigdata

Anonymous — Thu, 25 Oct 2018 05:33:22 GMT

Hi @xdshi,

I don't have any firewall or proxy and tried by restarting Talend studio multiple times.

I have already downloaded many third party libraries (both required and optional), so will the proxy/firewall affect rest of the libraries?

I have tried the same process in two laptops by using the same Talend version (latest), but no luck.

Re: unable to connect hadoop using Talend Bigdata

Anonymous — Thu, 25 Oct 2018 07:49:02 GMT

Hello,

Is there any issue with server availability(https://talend-update.talend.com/nexus/) from your side? Are you able to access the URL outside talend?

https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/hive-exec-1.2.0-mapr-1710/6.0.0/

You can copy and paste URL on your browser and check whether you are able to access it.

We will appreciate it a lot if you could post Error Log on forum

To show the Error Log view on the tab system, go to Window > Show views, then expand the General node and select Error Log.

Best regards

Sabrina

Re: unable to connect hadoop using Talend Bigdata

Anonymous — Thu, 25 Oct 2018 11:58:43 GMT

Hi @xdshi,

I am able to access those URLs which you provided.

I have downloaded around 70% of the libraries by disabling firewall but failed again and unable to download rest libraries.

Please refer the attached log file for error log.

error_log_final.log

Re: unable to connect hadoop using Talend Bigdata

manodwhb — Fri, 26 Oct 2018 05:29:59 GMT

@Vibin_CT,can you check the required jars are installed in your talend setup? in Module section you will find the all jars.

Re: unable to connect hadoop using Talend Bigdata

Anonymous — Fri, 26 Oct 2018 07:36:21 GMT

Hello,

Does the issue 'apache-curator' block you to use TOS-BD? It is not necessary to download all optional modules before using TOS-BD. The required jars will be pop-up again as a download/install dialog when you need to use some detailed function.

Here exists a jira issue about:

https://jira.talendforge.org/browse/TUP-20803

Best regards

Sabrina

Re: unable to connect hadoop using Talend Bigdata

Anonymous — Fri, 26 Oct 2018 07:47:39 GMT

Hi @manodwhb,

Could you please specify where exactly I can see installed jars in Talend?

Re: unable to connect hadoop using Talend Bigdata

Anonymous — Fri, 26 Oct 2018 07:53:35 GMT

Hi @xdshi,

Thanks for the information. I think 'Apache-curator' is blocking other jars to download.

But I am unable to see any jars needed for tHDFSConnection component for 'Custom-Unsupported' Distribution option.

Re: unable to connect hadoop using Talend Bigdata

Anonymous — Fri, 26 Oct 2018 08:10:52 GMT

Hello,

Did you follow up this online document about:TalendHelpCenter:Connecting to a custom Hadoop distribution to connect hadoop distribution 2.7.7?

Have you tried to connect hadoop with supported distribution by talend(2.5, 2.6) to see if it works?

Best regards

Sabrina

Re: unable to connect hadoop using Talend Bigdata

Anonymous — Fri, 26 Oct 2018 19:24:06 GMT

Hi @xdshi,

I tried for the custom distribution also, but still it is asking some libraries to download and unable to download.

At last I have installed Cloudera in VMware and tried same distribution in Talend. Now I am not getting any compile time error. It seems the issue is with custom distribution.

Re: unable to connect hadoop using Talend Bigdata

Anonymous — Mon, 29 Oct 2018 03:18:07 GMT

Hello,

@Vibin_CT wrote:

Hi @xdshi,

I tried for the custom distribution also, but still it is asking some libraries to download and unable to download.

At last I have installed Cloudera in VMware and tried same distribution in Talend. Now I am not getting any compile time error. It seems the issue is with custom distribution.

Thanks for your feedback. Is the connection hadoop 2.7.7 OK with you?

Best regards

Sabrina

Re: unable to connect hadoop using Talend Bigdata

Anonymous — Mon, 29 Oct 2018 06:06:22 GMT

Hi @xdshi,

Yes. Now I am able to connect Hadoop (Cloudera VMware) using Talend Big Data. Appreciate your valuable comments.