topic Re: Derived Context Variables in Talend Studio

Derived Context Variables

Anonymous — Thu, 04 May 2017 21:53:07 GMT

Hi Team,

Lets say I have a context variables var1 and var2.

var1="abc"

I need var2 to be var1+" def"

Both var1 and var2 are on the same context group. How can I achieve that?

Re: Derived Context Variables

TRF — Thu, 04 May 2017 22:03:16 GMT

Hi,
Just write context.var2 = context.var1 + " def" in a tJavaRow (or tJavaXxx) component.

Re: Derived Context Variables

Anonymous — Thu, 04 May 2017 22:05:17 GMT

Hi TRF,

I needed to use it inside the context group itself and assign to another
variable!
Is that possible?

Re: Derived Context Variables

TRF — Thu, 04 May 2017 22:11:58 GMT

No, you cannot have code in the place where context variable are defined. Dynamic values must come from tJavaXxx components. You may also define global variables in a PreJob wich is a good practice to achieve initialization tasks when job starts.

Re: Derived Context Variables

Anonymous — Thu, 04 May 2017 22:17:08 GMT

Thanks TRF.
To me it sounds inefficient. Just because Talend allows only literal values
over there, we have to do the overhead of writing all those prejob codes.

Re: Derived Context Variables

TRF — Thu, 04 May 2017 22:24:03 GMT

You have no choice

Re: Derived Context Variables

Anonymous — Fri, 05 May 2017 01:05:54 GMT

I don't understand what is inefficient about this. Maybe I am misunderstanding? The only limitation regarding this issue is that you cannot add code to be performed by the Context variables themselves, however you can write a job that can be used in ALL of your jobs to carry out this code. If you are happy writing Java you can perform this sort of functionality with Java Routines or a tJavaXXX (suggested by @TRF) as well. You can also store the context variables in a database and perform any config calculations on this data during import before the jobs start.

If you can elaborate on why this is problematic I believe that myself or @TRF can point you towards an elegant solution to your problem.

Re: Derived Context Variables

vapukov — Fri, 05 May 2017 01:18:04 GMT

@rhall wrote:

I don't understand what is inefficient about this. Maybe I am misunderstanding?...

It normal situation, when anybody try to implement logic in new tools exactly as it was somewhere else, or exactly as it designed

Similar questions regular like - "we do the same in Informatica (Mulesoft, ODI, Pentaho ... ) and want exactly this"

It wrong ways of course.

Unfortunately - not possible to give full answer without understanding expected logic, there are many ways (including suggested by You)

for example source idea:

var1="abc"
I need var2 to be var1+" def"

could be transformed to:

baseVar1="abc" in context group

and on feature steps in Job:
var2=baseVar+" def"

Re: Derived Context Variables

Anonymous — Fri, 05 May 2017 03:38:48 GMT

Hi @rhall

Thanks for your reply. Actually no it is not problematic and Thanks @TRF and Yourself for the suggested solutions.

Why I say it is inefficient is, we are setting up parameters on the context groups and therefore would expect them to be self contained and not require some other prejobs to complete their interpretation. What we are having to work with is workarounds, which I am happy to do if there are no other alternatives but should be at-least raised for future releases.

Coming to the ask, I would want to compute my data directories according to a particular flag that helps me have different working areas on the same server for parallel testing.

For e.g. /a/b/c/uat1/data and /a/b/c/uat2/data

Now the approaches I am able to think off are (Am Open to suggestions) :

1.) Different Context Groups for each uat1/uat2 -- I am not too keen on this, because for 5-6 parameters I will have to re-do all 300 odd parameters. More headache for the developer(unless there are any sleek shortcuts to do it) and also bulkier builds!

2.) Using Prejob with tJava components -- I might have to go with this approach. But the problem I can think off is, even for parameter interpretation, all the code is not in one place. We have our contexts that create placeholders and then again another job that actually interprets values, which makes it less than desirable for me.

P.S: I have switched to a different username, but still the same user.

Thanks

Rabi Panda

Re: Derived Context Variables

vapukov — Fri, 05 May 2017 04:18:08 GMT

for this I use only 1 context variable! 🙂

it name - context.env (I use 3 primary context group, but no limits at all)

prod - "prod"

qual - "qual"

dev - "dev"

in all other components (where it necessary ) all names concat like:

source_path == context.sourcefolder+"_"+context.env+"/data"

target_path == context.targetfolder+"_"+context.env+"/data"

and etc

same context.env used and run SubJob with dynamic context

When I need more than 3 - I just initialise context.env with individual value

Plus this variables used for load context values from csv file as part of filename

context.contextfilepath+"context_"+context.env+".csv"

and in any Job only 2 variables have value

- context.contextfilepath

- context.env

it give extremely flexibility

Re: Derived Context Variables

Anonymous — Fri, 05 May 2017 05:44:41 GMT

Hi @vapukov

Similar questions regular like - "we do the same in Informatica (Mulesoft, ODI, Pentaho ... ) and want exactly this"

It wrong ways of course.

Thanks for your reply.
Considering Talend is relatively new in the ETL space comparison is inevitable.
We could argue that comparing is wrong, I am of the opinion if something works for one should be scope for improvement for the other.

Regards,

Rabi Panda

Re: Derived Context Variables

Anonymous — Fri, 05 May 2017 05:48:43 GMT

@vapukov wrote:

for this I use only 1 context variable! 🙂

it name - context.env (I use 3 primary context group, but no limits at all)

prod - "prod"

qual - "qual"

dev - "dev"

in all other components (where it necessary ) all names concat like:

source_path == context.sourcefolder+"_"+context.env+"/data"

target_path == context.targetfolder+"_"+context.env+"/data"

and etc

same context.env used and run SubJob with dynamic context

When I need more than 3 - I just initialise context.env with individual value

Plus this variables used for load context values from csv file as part of filename
context.contextfilepath+"context_"+context.env+".csv"
and in any Job only 2 variables have value

- context.contextfilepath

- context.env

it give extremely flexibility

Yup we could do this, but then again I am sure you would agree, defining/computing the above expression at one place using it at the rest sounds a better approach.

Regards,

Rabi Panda

Re: Derived Context Variables

Anonymous — Fri, 05 May 2017 09:49:27 GMT

First of all, never use Contexts other than default. Many people confuse the Context terminology. A context is your set of context variables (DEV, TEST, PROD, default, etc). A Context Group is a batch of related context variables. A Context variables is....well you know that. Using multiple Contexts just adds risk to process. You can mitigate this if you want to have multiple environments by using flat files or a database with the Implicit Context Load. You can make the Implicit Context Load as simple or as complicated (and secure, tailorable, etc) as you like. It is briefly described here https://help.talend.com/display/TalendOpenStudioforDataIntegrationUserGuide60EN/C.3.4+Configuring+the+Implicit+Context+Load+feature

Re: Derived Context Variables

vapukov — Fri, 05 May 2017 10:31:47 GMT

may be not for 100% agree with - never/ever

life is different, so in some cases - it useful

in case when each job has it own context file - it easy to change file, but in case when many jobs share connection and other information ... better to have

TAC allow change single variable for single Job, with Community version it not true, so You will need or add command line parameter for point to different context file (database).

other case - my client work on windows server, I'm on Mac - file path will be different, so we have 2 Context at least, 1 point to my local context file, other point to his

on different projects - meet with all possible variant, but not sure which better

Re: Derived Context Variables

Anonymous — Fri, 05 May 2017 11:41:07 GMT

Many of those issues (file paths, for example) can be resolved with operating system Environment Variables. My current use of Context Variables only uses default, makes use of environment variables and encrypted db parameters and Java routines in the Implicit Context Load config (yes, routines work there) allowing a pretty secure but easily malleable Context system. My Prod and Test environments cannot be changed without access to the context db, while the developers can all share context settings or set their own. This is pretty complicated, but once configured all it requires is administration of the context db.

The reason I dislike Contexs (Prod, Test, Dev, etc) is that if you have child jobs being called by parent jobs (who doesn't) you can easily run into issues when the Contexts are set incorrectly. This doesn't need to even be thought about with the system I use.

Re: Derived Context Variables

vapukov — Fri, 05 May 2017 12:01:16 GMT

agree, Your way is good.

I also have experience with OS variables, working solution.

for now I choose as flexible for my needs, next way:

in tRunJob:

and in context:

Re: Derived Context Variables

Anonymous — Tue, 09 May 2017 12:58:54 GMT

Thanks @rhall

Trying to implement that.

However I am not able to reference the variables set using ImplicitCOntextLoad the same way we reference other contexts.

i.e. context.var1

Can you please help!

Re: Derived Context Variables

Anonymous — Tue, 09 May 2017 13:28:15 GMT

You have to set the context variables up in the Job. Have you done this? By that I mean simply either add your Context Group (if you have them grouped) or create each variable in the contexts section of your job. Once they exist, the implicit context load will bring them in.

Re: Derived Context Variables

Anonymous — Tue, 09 May 2017 13:38:37 GMT

In fact ignore the query, I had not defined it anywhere. Works for me now. Thanks for your help

Re: Derived Context Variables

Anonymous — Tue, 09 May 2017 13:53:10 GMT

No problem