AWS KMS Encryption Decryption in talend

[Sooraj1]

Hi, I have a use case where we are receiving the AWS KMS encrypted file in s3 and this file i have to decrypt in talend for data load. Has any one ever worked around it ?

Thanks

[quentin-vigne]

Hello @Sooraj1 

When using tS3Get and tS3Put component in Talend Studio there is an option called "Client-side Encrypt"

Use the value "KMS-managed customer master key" and fill in your key. Your objects are automatically encrypted and decrypted as part of your Amazon S3 PutObjects and GetObjects requests. 

tS3Get will Decrypt and tS3Put will Encrypt if using this option

Tell me if you still need more help

- Quentin

[Sooraj1]

Hi @quentin-vigne ,

Thanks for taking out time to reply on the above query, does this option work when we are reading the file from S3 which is KMS encrypted and needs to be decrypted in Talend to load the data? Because what i understand from the Talend documentation its used when we are trying to send data by encrypting using master key.

Please check the attached screenshot for same. I might be wrong though. Can you please confirm if that's the case?

Thanks

[quentin-vigne]

Yes, as I said above the KMS Encryption / Decryption used will be done on S3 side.

Depending of the component you are using, you're either going to decrypt the file or encrypt it.

tS3Get will Decrypt using KMS

tS3Put will Encrypt 

Give it a try and you'll see

Don't forget to mark the post as solved if everything is ok for you 

- Quentin

[Sooraj1]

Also, i want to highlight that this source file which is being uploaded in S3 with KMS encryption is zip file which when extracted provides 2 files, one is CSV (Data file), and one is Key file. So idea here is to decrypt the data (CSV) file using Key file. I am not sure how ts3get will decrypt the zipped file. Can you please give solution to this problem statement please.

[quentin-vigne]

Here’s how I would approach it:

1. First, use tS3Get to retrieve the file.

2. Then, use an "if" trigger with the following condition:

((String)globalMap.get("tS3List_1_CURRENT_KEY")).endsWith(".gz")

(Replace "gz" with "zip" or any other compression method you use.)

This will allow you to unarchive the file.

• Next, read the key file normally with a tFileInputDelimited and store its value in a context variable. Let’s call it "kms_value".

• Finally, use tS3Get again, and in the "Client-side Encrypt" option:

  • Select "KMS-managed customer master key"

  • Set the value to "context.kms_value"

Since you're retrieving the key before using it, this approach should work without problem.

- Quentin

[Sooraj1]

Hi Quentin,

Thanks for taking out time to explain each step in detail, i tried the above solution but getting below error in the final ts3get component :

[WARN ] 11:39:07 com.amazonaws.services.s3.internal.crypto.v1.S3CryptoModuleAE- Unable to detect encryption information for object '4ece0230b5ed7778b49034d4a7968ddc.zip' in bucket 'onboardingdatastack-securebucket6257cae8-m7on2iyvve0w'. Returning object without decryption.

i am able to get the file from s3 unzip it and then trying to decrypt but getting above error.

Here is my job flow :

 and tfileinputdelimited propertoes : 

these are the files after unzipping :

Can you please tell me what's wrong here? Please let me know if you need more info.

Really appreciate your replies, Kudos 🙂

[quentin-vigne]

Hi @Sooraj1 

The first part looks OK, what I want to see is what did you use for tS3Get encryption value ?

You should probably add a tJavaRow component to get the value of your encryption key from the tFileInputDelimited to a context variable.

First in the Context menu create a new variable, call it for example "kms_value" or whatever then in the tJavaRow :

context.kms_value = row1.column

row1 being the name of the row and column the name you are using

- Quentin

[Sooraj1]

Hi Quentin,

i am passing context.kms_value as suggested by you in ts3get encryption value.

Also i tried using tjavarow but still same error, attaching snapshots for your reference :

[quentin-vigne]

You need to put the context without "" 

context.kms_value

and not

"context.kms_value"

- Quentin