topic Re: multiple implicit context loads from multiple files in Talend Studio

multiple implicit context loads from multiple files

n999 — Sat, 16 Nov 2024 07:12:49 GMT

When you open a project you can set the project level implicit context load to read from a single file.

I was wondering if you can be more fine grained than this if you want to keep context values in different files.

Project level implicit context load only allows you to load from one file.

I was wondering how implicit context load works when you have reference projects.

Can you specify implicit context load in each reference project with different file paths in each reference project?

So reference project1 implicit context loads from file1 and reference project2 implicit context loads from file2.

Then the parent project could reference both reference projects and implicitly load context variables from both files?

Does Talend support this?

Is there another way of using multiple files?

thanks

Re: multiple implicit context loads from multiple files

Anonymous — Mon, 26 Nov 2018 11:53:09 GMT

You can be clever here. A while ago I discovered that you can use Routines to configure your Implicit Context Load parameters. What I have done is configure my contexts to be loaded with different values depending upon the machine they are run on. I have set up System Environment Variables to identify the environment, etc, and that tells my ImplicitContextLoad Routine (a routine I wrote) where to get my contexts from. It's actually really simple. To see how it works set up a Routine for yourself. Just configure simple Methods to return hardcoded values for your Implicit Context Load params. Use these Routine Methods in your Implicit Context Load settings. Once you see it working, you can experiment to suit your requirements exactly. This hidden functionality opens A LOT of doors.

Re: multiple implicit context loads from multiple files

n999 — Mon, 26 Nov 2018 12:09:46 GMT

That sounds interesting thanks for that.

"Use these Routine Methods in your Implicit Context Load settings." - how would I do this?

Project level Implicit Context Load settings only allow file or db use?

Cheers

Re: multiple implicit context loads from multiple files

Anonymous — Mon, 26 Nov 2018 12:38:16 GMT

The parameters that you set for file or db use you populate with your new Routine. For example, my Implicit Context Load config is setup as "From Database". My "Host" parameter has this code....

routines.ImplicitContextLoadUtils.getImplicitContextParameterValue("TalendContextHost",rootPid,pid)

..., my "Db Name" parameter has this code....

routines.ImplicitContextLoadUtils.getImplicitContextParameterValue("TalendContextDbName",rootPid,pid)

,etc. Ignore the actual methods as these are specific to my requirements, but these are methods NOT literal values or context variables I am using. This means that when the job runs, the routine Methods are called first, they return the appropriate values to log in to the database I want the context variables from and then everything else runs.

Re: multiple implicit context loads from multiple files

n999 — Wed, 28 Nov 2018 12:01:59 GMT

awesome!

I know you could get access to context variables in those settings but being able to call routines is news to me! (Hope that functionality remains )

Many thanks for that. Will give that a go. Might test the reference project implicit context loads as well to see if that works.

Cheers

Re: multiple implicit context loads from multiple files

n999 — Wed, 05 Dec 2018 12:59:59 GMT

Reference project implicit context loads for reference project do not happen. you can only have one project implicit context load, the parent project...

So I'm looking further at calling routines in those fields.

We want to encrypt our DB context vales and have the select query use a key in the select statement to decrypt these values..(not as good as talend using a key to decrypt the value at run time but anyway...)

Can you add routines in the "Table Name" and "Query Condition" implicit context load fields?

Cheers

Re: multiple implicit context loads from multiple files

Anonymous — Wed, 05 Dec 2018 14:11:56 GMT

You can have routines called in all Implicit Context Load parameters.

For encryption take a look at JASYPT (http://www.jasypt.org/). It allows you to use encrypted values in a file, etc, simply by wrapping by wrapping them in ENC( ) tags. I use this functionality as well.

Re: multiple implicit context loads from multiple files

n999 — Tue, 21 May 2019 15:07:28 GMT

Still deciding over files vs database and read your excellent post at https://www.talend.com/blog/2019/02/28/best-practices-for-using-context-variables-with-talend-part-3/

I am concerned about having to store context values that are credentials in plain text in any db table that the implicit context load reads...At least with a file any sensitive context values sit only on the server.

Would be good if the values in the context table could be encrypted/decrypted too

Cheers

Nomit

Re: multiple implicit context loads from multiple files

Anonymous — Sun, 26 May 2019 09:47:50 GMT

Hi @n999,

I cover this in my second blog on Context Variables. I had meant to write it all into 1 big blog, but thought that it would be better to break it down. The link to the second blog is here https://www.talend.com/blog/2019/02/19/best-practices-for-using-context-variables-with-talend-part-2/

I use JASYPT to encode password fields in text files (it could also be used for database values).

I hope this helps.

For my other blogs, just click on my name at the top of the blog 🙂

Re: multiple implicit context loads from multiple files

n999 — Mon, 03 Jun 2019 12:29:51 GMT

Hi - Thanks for responding.

"I use JASYPT to encode password fields in text files (it could also be used for database values)."

I can see how this would work with a text file but not with database values. Talend requires a table with "key" and "value" for the columns and these are selected by a hidden query you can't edit. How would you

1) either add JASYPT into this select statement or

2) pass the selected values through a routine using JASYPT?

Cheers

Re: multiple implicit context loads from multiple files

Anonymous — Mon, 03 Jun 2019 12:43:06 GMT

Since passwords are Strings (or can be if you are encrypting them), you can simply extrapolate from the following code as an example of how you would encrypt/decrypt those Strings.

import org.jasypt.encryption.pbe.StandardPBEStringEncryptor;

//Configure encryptor class

StandardPBEStringEncryptor encryptor = new StandardPBEStringEncryptor();

encryptor.setAlgorithm("PBEWithMD5AndDES");

encryptor.setPassword("BOB");




//Set the String to encrypt and print it

String stringToEncrypt = "Hello World";

System.out.println(stringToEncrypt);




//Encrypt the String and store it as the cipher String. Then print it

String cipher = encryptor.encrypt(stringToEncrypt);

System.out.println(cipher);




//Decrypt the String just encrypted and print it out

System.out.println(encryptor.decrypt(cipher));

The Implicit Context Load will load the Context Variables behind the scenes. To alter the encrypted values, all you need to do is place a tPreJob into your job with a tJava component attached. You can then decrypt there. The rest of the job will start after this has been done.

Re: multiple implicit context loads from multiple files

n999 — Mon, 03 Jun 2019 12:47:37 GMT

Ahh okay - I'd get a handle on the encrypted values in a pre-job.

cheers for all your help!

Re: multiple implicit context loads from multiple files

Anonymous — Mon, 03 Jun 2019 13:29:02 GMT

No problem at all 🙂

Re: multiple implicit context loads from multiple files

LGTLEON — Thu, 02 Feb 2023 11:06:47 GMT

Congrats for your article mentioned above. It is really helpful! I have one question: I am using Implicit context loading from file. How could I load the context only in parent jobs? I Know you did it using the "where" clause and rootId, jobID variables, but loading from file, a FileNotFoundException raises if a correct file path is not given. Thanks!

Re: multiple implicit context loads from multiple files

Anonymous — Thu, 02 Feb 2023 16:40:36 GMT

If I understand you correctly, you will have a parent job which contains several child jobs. You only want to use the implicit context load in the parent. Is that correct? If so, you can do the following.

Look at this job tab below.....

You will see that the "Use Project Settings" tickbox is ticked. Untick this and you will be able to set the "Implicit tContextLoad" tickbox. You will see this when it is done....

Here you can set up your parameters.

In your child jobs, make sure that the Implicit Context Load is not set.

Re: multiple implicit context loads from multiple files

LGTLEON — Fri, 03 Feb 2023 11:38:50 GMT

Thanks for your quick answer.

After several days of reading many articles about Talend context I am really confused about which of these approaches is more suitable.

A) Implicit context load from file in all jobs (parent and child). Context variables values in child jobs will not be overridden by values from parent job unless they don´t exist in the external file. https://community.talend.com/s/article/Implicit-Context-Load-usage-in-both-parent-and-child-jobs-0GoRi?language=en_US

B) Implicit context load from file only in parent jobs. I must add all context variables in parent job and pass them to sub-jobs through Transmit whole context.

Considering your experience ¿which one would you advise? Would be there any difference in terms of eficciency?

I would be really gratefull if you could help me to clarify this issue.

Re: multiple implicit context loads from multiple files

Anonymous — Fri, 03 Feb 2023 17:27:35 GMT

First of all, it very much depends on what you intend to do with Talend. If you are working with Talend Cloud, it might make more sense to use the context variable configuration built in to that. It can certainly aid with ensuring your whole team are working with the same variables. But if you are using Talend Open Studio, you don't need to worry about that.

With regard to the choices above, I would probably go with Implicit Context Load for the parent jobs only. You have to make sure you set up the child jobs to receive the context values from the parent though.

Re: multiple implicit context loads from multiple files

LGTLEON — Mon, 06 Feb 2023 12:56:24 GMT

Thanks again for your help.

I´m using Talend Open Studio and I export my jobs as executable batch files. Considering the second option: is it recommended to use Transmit whole context option to send all of my context variables to the child jobs? or is it a bad practice that causes the programming to be heavy?

Re: multiple implicit context loads from multiple files

Anonymous — Mon, 06 Feb 2023 14:16:36 GMT

You can manually specify the context variables that are sent to the child job if you prefer. Clicking the option to send everything is just quicker and requires less maintenance.

Re: multiple implicit context loads from multiple files

LGTLEON — Wed, 08 Feb 2023 10:18:37 GMT

Thanks again. I really appreciate your help.