Unable to process(copy) the data on hive with S3 External Table

Anonymous — Sat, 16 Nov 2024 06:43:37 GMT

Hi there.

Now we are working on EMR(Spark) - Talend BigData Platform PoC.

We would like to copy the data from one S3 external hive table to another S3 external hive table with BigData Platform.

(Both of the external table are on EMR)

After creating the external table on hive, we tried to copy the data through the Talend Studio.

Actually, the data would be processed with Spark, but failed.

19/01/30 05:35:49 WARN ipc.Client: Failed to connect to server: ip-10-118-120-67.ap-northeast-1.compute.internal/10.118.120.67:8020: try once and fail.
java.net.NoRouteToHostException: No route to host

Please see the attached log.

It seems that the Talend could not connect to the nodes.

We have only 2 nodes(master and slave) .

The IPs are 10.118.120.90 and 10.118.120.114, not 10.118.120.67.

Sorry, I moved this topic to the design and development topic.

The host with ip-10-118-120-67.ap-northeast-1.compute.internal is the previous node's ip.

We always stop the EMR cluster in the midnight and start the cluster in the morning because of its cost.

We are not sure why Talend(job) is trying to connect to 10.118.120.67..

We surely changed the configuration like cluster, hdfs, db connection, remote server(we installed the job server on EMR master node..).

Can anyone give us advise for this issue??

Thanks.

topic Unable to process(copy) the data on hive with S3 External Table in Talend Studio

Unable to process(copy) the data on hive with S3 External Table