S3 credentials based on environment

[danblo]

Our job servers (QA and Prod) reside on EC2  instance in AWS.  I have set up a policy for EC2 instances to access S3 without the need to use the keys.  So in the talend components accessing S3 I can use Inherit credentials from AWS role option.  However that is not available in our local dev environments where we have the studio installed.   On our local machines we have to use keys that each developer gets.  How can we set up talend job to use different authentication method based on environment where the job is executing from.

[Anonymous]

Hello,

Have you tried to add different environments to context group & propagate context change to your job?

Please refer to online user guide about:TalendHelpCenter:Centralizing context variables in the Repository

Let us know if it is what you are looking for.

Best regards

Sabrina

[Anonymous]

I'm facing the same problem now, did you ever find a solution that worked?

@xdshi - The problem is that we need the s3 component to behave differently depending on context, not just use different variables. If you tell the component to inherit credentials from its IAM role it will attempt to contact the EC2 metadata service, which will not exist when not on an ec2 machine. Is there a way to use context to have the s3 component inherit credentials from the IAM role in one case, and use access keys in another?

[Anonymous]

Hello @kheuton,

---

@kheuton wrote:

I'm facing the same problem now, did you ever find a solution that worked?

@xdshi - The problem is that we need the s3 component to behave differently depending on context, not just use different variables. If you tell the component to inherit credentials from its IAM role it will attempt to contact the EC2 metadata service, which will not exist when not on an ec2 machine. Is there a way to use context to have the s3 component inherit credentials from the IAM role in one case, and use access keys in another?

---

"Inherit credentials from AWS role" is to obtain AWS security credentials from Amazon EC2 instance metadata. To use this option, the Amazon EC2 instance must be started and your Job must be running on Amazon EC2.

For more information, see Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances

Best regards

Sabrina

Best regards

Sabrina

[Anonymous]

It is in general not a good idea to let a job behave differently depending the stage (test, prod etc) because this way you would be unable to use the unchanged job moving from one stage to another stage.