topic Re: Read S3 file in Talend Spark Job in Talend Studio

Read S3 file in Talend Spark Job

Anonymous — Sat, 16 Nov 2024 07:52:52 GMT

error:

Checking ports...

Sending job 'Framework_Semantic_Abac' to server (10.1.6.23:8001)...

File transfer completed.

Deploying job 'Framework_Semantic_Abac' on server (10.1.6.23:8000)...

Running job 'Framework_Semantic_Abac'...
Starting job Framework_Semantic_Abac at 14:18 30/07/2018.

[statistics] connecting to socket on port 3954
[statistics] connected
[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[WARN ]: org.apache.spark.SparkConf - In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
[statistics] disconnected
[ERROR]: semantic_layer_se.framework_semantic_abac_0_1.Framework_Semantic_Abac - TalendJob: 'Framework_Semantic_Abac' - Failed with exit code: 1.
java.lang.IllegalArgumentException: Invalid hostname in URI s3n:

Re: Read S3 file in Talend Spark Job

Anonymous — Mon, 06 Aug 2018 07:57:47 GMT

Hello,

Did your secret key for S3 contains a "/" an invalid hostname?

Could you please refer to this issue page to see if it is what you are looking for?

https://issues.apache.org/jira/browse/HADOOP-3733?focusedCommentId=15132075&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15132075

Best regards

Sabrina

Re: Read S3 file in Talend Spark Job

adolitos — Mon, 27 Jan 2020 12:53:46 GMT

Hi, It seems this issue is not solved.

I read the details on the jira issue link (https://issues.apache.org/jira/browse/HADOOP-3733?focusedCommentId=15132075&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15132075)

1) Yes, the guys there said to encode "/" and "+" characters, but even doing this the connection have not worked.

2) According to the further comments it's not ideal passing credentials on URI, they advice to avoid this if possible... But I think this is not possible, this would be possible if the tS3Configuration get changed, manipulating the key_id and secret_key in a different way.

3) Generating a key_id and secret_key until getting values without "/" is a workaround... but I think this is not acceptable.

Could you give a help with that?