tS3Configuration in BigDataBatch Job is using s3a file system to enable inherit credentials from AWS role - is there any option to use S3N file system with inherit credentials option ?

[Anonymous]

Hello:

tS3Configuration in BigDataBatch Job is using s3a file system to enable inherit credentials from AWS role.

Is there any option to use S3N file system with inherit credentials option i.e; that is without providing access Key and secret Key ?

Problem is :

when we try to use tRedshiftConfiguration , it is required to provide tS3Configuration and when we select - use s3a file system, getting some access issues while running spark job in EMR  using s3a file system option checked in talend s3configuration job.

Error:

java.nio.file.AccessDeniedException: s3a://<<Location>>/_temporary/0: innerMkdirs on s3a://<<location>>/_temporary/0: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied;

Please reply - if anyone performed any redshift operations in spark job using s3a file system in s3configuration.

Thanks

Vijay

[Anonymous]

Hello,

"Inherit credentials from AWS role" option is available in Big Data Spark Job (EMR) in 7.2.1, 7.1.2.

This means that it is not required to input access/secret key in tS3Configuration in Spark Batch and Streaming jobs.

Behind the scene the authentication used is the one of the EMR cluster / role so the job would have the same rights as the EMR cluster.

Best regards

Sabrina