topic Re: AWS KMS Encryption Decryption in talend in Talend Studio

AWS KMS Encryption Decryption in talend

Sooraj1 — Wed, 02 Apr 2025 10:51:00 GMT

Hi, I have a use case where we are receiving the AWS KMS encrypted file in s3 and this file i have to decrypt in talend for data load. Has any one ever worked around it ?

Thanks

Re: AWS KMS Encryption Decryption in talend

quentin-vigne — Wed, 02 Apr 2025 11:57:58 GMT

Hello @Sooraj1

When using tS3Get and tS3Put component in Talend Studio there is an option called "Client-side Encrypt"

Use the value "KMS-managed customer master key" and fill in your key. Your objects are automatically encrypted and decrypted as part of your Amazon S3 PutObjects and GetObjects requests.

tS3Get will Decrypt and tS3Put will Encrypt if using this option

Tell me if you still need more help

- Quentin

Re: AWS KMS Encryption Decryption in talend

Sooraj1 — Thu, 03 Apr 2025 07:47:47 GMT

Hi @quentin-vigne ,

Thanks for taking out time to reply on the above query, does this option work when we are reading the file from S3 which is KMS encrypted and needs to be decrypted in Talend to load the data? Because what i understand from the Talend documentation its used when we are trying to send data by encrypting using master key.

Please check the attached screenshot for same. I might be wrong though. Can you please confirm if that's the case?

Thanks

Re: AWS KMS Encryption Decryption in talend

quentin-vigne — Thu, 03 Apr 2025 07:44:47 GMT

Yes, as I said above the KMS Encryption / Decryption used will be done on S3 side.

Depending of the component you are using, you're either going to decrypt the file or encrypt it.

tS3Get will Decrypt using KMS

tS3Put will Encrypt

Give it a try and you'll see

Don't forget to mark the post as solved if everything is ok for you

- Quentin

Re: AWS KMS Encryption Decryption in talend

Sooraj1 — Thu, 03 Apr 2025 07:50:17 GMT

Also, i want to highlight that this source file which is being uploaded in S3 with KMS encryption is zip file which when extracted provides 2 files, one is CSV (Data file), and one is Key file. So idea here is to decrypt the data (CSV) file using Key file. I am not sure how ts3get will decrypt the zipped file. Can you please give solution to this problem statement please.

Re: AWS KMS Encryption Decryption in talend

quentin-vigne — Thu, 03 Apr 2025 12:06:31 GMT

Here’s how I would approach it:

First, use tS3Get to retrieve the file.
Then, use an "if" trigger with the following condition:

((String)globalMap.get("tS3List_1_CURRENT_KEY")).endsWith(".gz")

(Replace "gz" with "zip" or any other compression method you use.)

This will allow you to unarchive the file.

Next, read the key file normally with a tFileInputDelimited and store its value in a context variable. Let’s call it "kms_value".
Finally, use tS3Get again, and in the "Client-side Encrypt" option:
- Select "KMS-managed customer master key"
- Set the value to "context.kms_value"

Since you're retrieving the key before using it, this approach should work without problem.

- Quentin

Re: AWS KMS Encryption Decryption in talend

Sooraj1 — Fri, 04 Apr 2025 06:16:00 GMT

Hi Quentin,

Thanks for taking out time to explain each step in detail, i tried the above solution but getting below error in the final ts3get component :

[WARN ] 11:39:07 com.amazonaws.services.s3.internal.crypto.v1.S3CryptoModuleAE- Unable to detect encryption information for object '4ece0230b5ed7778b49034d4a7968ddc.zip' in bucket 'onboardingdatastack-securebucket6257cae8-m7on2iyvve0w'. Returning object without decryption.

i am able to get the file from s3 unzip it and then trying to decrypt but getting above error.

Here is my job flow :

and tfileinputdelimited propertoes :

these are the files after unzipping :

Can you please tell me what's wrong here? Please let me know if you need more info.

Really appreciate your replies, Kudos 🙂

Re: AWS KMS Encryption Decryption in talend

quentin-vigne — Fri, 04 Apr 2025 08:09:01 GMT

Hi @Sooraj1

The first part looks OK, what I want to see is what did you use for tS3Get encryption value ?

You should probably add a tJavaRow component to get the value of your encryption key from the tFileInputDelimited to a context variable.

First in the Context menu create a new variable, call it for example "kms_value" or whatever then in the tJavaRow :

context.kms_value = row1.column

row1 being the name of the row and column the name you are using

- Quentin

Re: AWS KMS Encryption Decryption in talend

Sooraj1 — Fri, 04 Apr 2025 08:52:55 GMT

Hi Quentin,

i am passing context.kms_value as suggested by you in ts3get encryption value.

Also i tried using tjavarow but still same error, attaching snapshots for your reference :

Re: AWS KMS Encryption Decryption in talend

quentin-vigne — Fri, 04 Apr 2025 09:22:47 GMT

You need to put the context without ""

context.kms_value

and not

"context.kms_value"

- Quentin

Re: AWS KMS Encryption Decryption in talend

Sooraj1 — Fri, 04 Apr 2025 13:47:27 GMT

Tried running using

context.kms_value

But still same error.

Re: AWS KMS Encryption Decryption in talend

quentin-vigne — Fri, 04 Apr 2025 14:15:59 GMT

From your earlier screenshot I see you are passing 5 values in your tJavaRow component

What does your file contains ? If it contains various rows it's only going to use the last value of your file as encryption key

You need to treat each file individually especially if each one of them has it's own encryption key, because you're risking using the wrong encryption key to decrypt the file

It looks like you're on the right way for you job, if you want just use 1 file for the moment and try to add tLogRow to monitor the variable passing by and make sur it's the right value.

- Quentin

Re: AWS KMS Encryption Decryption in talend

Sooraj1 — Mon, 07 Apr 2025 08:24:03 GMT

Hi Quentin,

i am using only one file to decrypt the csv file using the key file (which supposed to have the key), you are right i am getting 5 rows from that file. i am also not sure why we are getting 5 rows in the key file. i am attaching the screenshot of the key file data :

Re: AWS KMS Encryption Decryption in talend

Sooraj1 — Mon, 07 Apr 2025 08:38:45 GMT

It seems like the key file is also encrypted which need to be converted to plaintext first and then used for decrypting the CSV file, Any idea how we can implement it in talend job?