topic Re: Data masking and anonymization using Talend in Talend Studio

Data masking and anonymization using Talend

Anonymous — Wed, 17 Apr 2013 10:11:16 GMT

I have a requirement to migrate data from a source database to a destination database. Also, we want to make sure some of the confidential data like passwords, phone numbers etc., should not be exposed in the destination database. We are planning to use talend data integration tool for this requirement, but I couldn't find out any information related to data masking in user manual.
Could someone please let me know if we can perform data masking/anonymizing using talend?
Thanks in advance for any help you are able to provide.
-Satish

Re: Data masking and anonymization using Talend

Anonymous — Wed, 17 Apr 2013 12:20:30 GMT

Hi Satish
No, you can't use Talend Studio to perform data masking, it is an ETL tool, not a data masking tool.
Shong

Re: Data masking and anonymization using Talend

Anonymous — Wed, 17 Apr 2013 19:45:51 GMT

As highlighted by Shong, there is no dedicated masking functionality. However, there are ways of achieving a similar result using what is available within Talend. For example replacing values with a random or. Lookup/replace - to maintain consistency - using the map component.
However, do also recognize that it would be easier if the transformation is done pre or post migration. Perhaps even using a dedicated tool where the data modeling can be done more elaborately.

Re: Data masking and anonymization using Talend

Anonymous — Thu, 18 Apr 2013 03:58:23 GMT

Thanks Shong and Kootstra for your valuable suggestions.

Re: Data masking and anonymization using Talend

Anonymous — Sun, 23 Jun 2013 06:40:54 GMT

Talend Open Studio is one of the best tools I have used in recent years but for the purpose of apply data masking I have been using Data Masker.
I do however agree with Kootstra_a that replacing values with a random or. Lookup/replace - to maintain consistency - using the map component is an inexpensive approach it wont always stand up in you test use cases.
Check out http://www.datakitchen.com.au/ for info about data masking, data masking methodology, data discovery and test data management.

Re: Data masking and anonymization using Talend

Anonymous — Mon, 24 Jun 2013 16:54:11 GMT

the key suggestion by the 'data kitchen' (i loved their photo - very over the top)
anyhow, they discuss 'Developing a Data Masking Methodology'
which can be achieve independently of the tool at hand.
so define your requirements, research and then use Talend to deliver it.
i have recently implemented similar workstream for a client and it works fine with Talend.
regards,

Re: Data masking and anonymization using Talend

Anonymous — Thu, 26 Sep 2013 15:10:32 GMT

Hii,
Can somebody explain how to have my job run with different schema dynamically?
Means my job have some tfileinput written to tlogrow and I am passing the file name to the tfileinput delimited thru the context variable. I want to display different file contents every time I run this job by passing different file name with different schema to the context variable.
How can I achieve that just by changing the file name provided in the context variable?
P.S:The different file names have different schemas.

Re: Data masking and anonymization using Talend

_AnonymousUser — Thu, 06 Mar 2014 12:37:31 GMT

Data Masking is a very complex topic actually. It has a lot of IP around it, thus the more-less proofed solutions are pretty expensive. There are simple ways to obfuscate data in ETL, but for some of you guys - would you be interested to look at the components suite that you can call from Talend via simple API?

Re: Data masking and anonymization using Talend

Anonymous — Mon, 06 Jul 2015 10:06:00 GMT

Hi,
I wonder if somebody did a tutorial on data masking using Talend. A lot of people is talking in general terms, but a detailed tutorial would be most appreciated.
Thanks in advance

Re: Data masking and anonymization using Talend

Anonymous — Mon, 06 Jul 2015 10:36:37 GMT

Hi spr655,

Here is a KB article about: TalendHelpCenter:How to setup encryption of the passwords in Talend Studio?
In addition, you can use tContextLoad or implicit tContextLoad to hide the password characters assigned to different context variables.
Best regards
Sabrina

Re: Data masking and anonymization using Talend

Anonymous — Fri, 17 Jul 2015 10:12:29 GMT

Dear Sabrina,
Thanks for the input. It is very interesting for future purposes. At the present time, my question is about how to replace values from input tables/files randomly with values of fake data tables. I have been checking possibilities with Talend, but I don't have that much experience with the product, so I didn't find a concrete solution.
The scenario is the following:
- in input I have 2 different tables and one xml file that contain the fields id, first name, surname and date of birth.
-I have a table that is populated with invented ids, first names, surnames and dates of birth
With Talend, for the same id (2 tables and the xml), I obtain the same fake values in the output (2 tables and the xml) with the help of components or routines.
I have been looking at possible solutions and I guess the java routines will play an important role in this kind of job, because I didn't or couldn't find a specific component/s that would achieve my purpose.
Could you help me with this?
Thanks in advance

Re: Data masking and anonymization using Talend

Anonymous — Mon, 20 Jul 2015 08:52:05 GMT

Hi,

The scenario is the following:
- in input I have 2 different tables and one xml file that contain the fields id, first name, surname and date of birth.
-I have a table that is populated with invented ids, first names, surnames and dates of birth
With Talend, for the same id (2 tables and the xml), I obtain the same fake values in the output (2 tables and the xml) with the help of components or routines.

Could you please elaborate your case with an example with input and expected output values?

Best regards
Sabrina

Re: Data masking and anonymization using Talend

Anonymous — Mon, 20 Jul 2015 11:10:51 GMT

Hi,
My post was treated as spam, so I had to upload it as images.
Thanks in advance for your help and support.

Re: Data masking and anonymization using Talend

Anonymous — Tue, 21 Jul 2015 08:57:11 GMT

Hi,
Would you mind taking a look at a use case about Implicit tContextLoad in this forum: https://community.talend.com/t5/Design-and-Development/resolved-Iterate-through-a-list-of-URL-s-for-tRSSInput/td-p/94539 to see if it is satisfying your needs?
Best regards
Sabrina

Re: Data masking and anonymization using Talend

Anonymous — Tue, 08 Sep 2015 20:13:12 GMT

Hello Everyone,
I saw a lot of suggetions, but does anyone has put in practice some example of Data Masking with Talend? I am trying to read and understand how Data Masking works to implement with Talend.
I really appreciate any help
Regards

Re: Data masking and anonymization using Talend

tYrannoSaurusRex_1 — Tue, 21 Feb 2017 08:53:27 GMT

You cannot expect Talend to identify email Addresses and phone nos. automatically to mask them. But you can mask specific keywords using tReplaceList.
Let's say you have comments column in your file wherein you want users to hide specific keywords likes Account, Phone, Email or Password. It can be done.

Re: Data masking and anonymization using Talend

Anonymous — Sat, 03 Nov 2018 16:13:24 GMT

Hello,
I am really new to the Talend open studio version 7.0 technology, and I am currently working on an "anonymization" project for the tables. In my STD they ask me to anonymize on several tables.
In the management rules
- Replace the string by 5 characters randomly generate followed by the string "archives"
ISO code case: Replace the string with "FR"
Case of the email: Replace the chain of characters email by "archives@archives.com"
IBAN code case: replaces the string with: "BL679999999999999999999934"
I do not know how that component does the job.
thank you for your collaboration I would be very grateful
Thank you