topic Re: [resolved] Configuring job to run on different environments in Talend Studio

[resolved] Configuring job to run on different environments

satyavar — Fri, 14 Nov 2014 07:27:56 GMT

Hi
I have developed around 60 jobs which are all pointing to my DEV environment. The Input and Output components are using Property Type as "Repository".
Now, I would like to execute all these jobs in STAGING and PRODUCTION environments? Please help me to understand on how can i make the same job pointing to different environment database connections depending on situation. How to configure metadatas?
Best regards,
Kalyan

Re: [resolved] Configuring job to run on different environments

Anonymous — Fri, 14 Nov 2014 08:04:53 GMT

Hi Kalyan,
Quite a pickle ....but Talend does provide for such situations
Selection 'Edit Connection' & move to step 2. Then, you can choose the below highlighted option & export your connection as contexts & propagate the change (in Property Type) to all affected jobs. Then, assign that context group to your jobs.
Once this setup is done, you can easily provide the environment specific values to the job via (projectname.properties) file & run them conveniently.
Hope this helps.
MathurM

Re: [resolved] Configuring job to run on different environments

satyavar — Fri, 14 Nov 2014 10:17:11 GMT

Thank you that helps. Just wondering how do i store the password in context variable? Looks like it is encrypted by TALEND. Also please let me know if i should create metadata's for each environment or is it just sufficient if i create metadata for one environment and just change the context variables accordingly for each environment?

Re: [resolved] Configuring job to run on different environments

Anonymous — Fri, 14 Nov 2014 11:19:25 GMT

The step that I showed in my screen-shot will only convert the current DB connection metadata into contexts. This will also create a context group by the name of the connection (which will get applied to the jobs, once you approve for that option).
Now in the connection, in place of each detail a context will be present whose value would be expected at runtime, including password. This values will be expected in the config file (projectname.properties file)

Re: [resolved] Configuring job to run on different environments

Anonymous — Fri, 14 Nov 2014 11:23:42 GMT

And all these objects, the job + the connection metadata + the context group, will always be bundled together whether you run your job from studio or you export it.
So, NO. You would not have to create separate metadata as per environments. Just maintain (switch) diff context values in the config file as per the environment you want to run your job in.

Re: [resolved] Configuring job to run on different environments

satyavar — Fri, 14 Nov 2014 14:49:52 GMT

Thanks for your reply.
Attached is the screenshot in which i have marked a black line for each context variable. Can someone please let me know how do i load values in to the context variables depending on the environment - Default, STG and PROD and from where should i load them? I am hardcoding the values for each of them but thought that might not be a good practice. Please let me know if there is any sample job which i can refer to.
There are around 60+ jobs in my studio, so should i keep loading the context variables everytime when a job runs?
Thanks
Kalyan

Re: [resolved] Configuring job to run on different environments

Anonymous — Mon, 17 Nov 2014 08:12:07 GMT

Hi Kalyan,
I can see you have configured 3 different value sets for each of your environment.
I'm interested to know, how would you switch between these at runtime? (even if manually, HOW??)
MathurM

Re: [resolved] Configuring job to run on different environments

satyavar — Mon, 17 Nov 2014 11:21:11 GMT

So, I have to keep only one column and load those values from a text file? But how will i load the password? It seems to be stored as an encrypted value. Should i store the encrypted value in my text file?

Re: [resolved] Configuring job to run on different environments

satyavar — Mon, 17 Nov 2014 11:48:48 GMT

I do not have access to create a metadata for Staging and PROD, so i cannot extract the metadata as a context for these 2 environments.

Re: [resolved] Configuring job to run on different environments

Anonymous — Mon, 17 Nov 2014 14:46:51 GMT

Nah, I just wanted to know that how are you planning to switch between the 3 sets (if at all, u had planned for that)

Re: [resolved] Configuring job to run on different environments

satyavar — Mon, 17 Nov 2014 14:48:14 GMT

No i did not plan to switch, I thought when my job runs on STAGING - it picks from STAGING contexts so on...

Re: [resolved] Configuring job to run on different environments

Anonymous — Mon, 17 Nov 2014 14:53:58 GMT

Ideally, you should prepare your job with contexts, & have it deployed in any environment like that.
Now, at runtime, the actual variable values should be furnished via a config file. As a result, there will be 3 diff config files (with persons with appropriate clearances having access to these files, including passwords)
so, 1 job + 3 config files, should solve your problem.
Let me know if this helps.
MathurM

Re: [resolved] Configuring job to run on different environments

satyavar — Mon, 17 Nov 2014 14:55:40 GMT

Yes i understand that there will be one config file each per environment. But how do i store the passwords in the file? Because i see that TALEND stores them in an encrypted format.

Re: [resolved] Configuring job to run on different environments

Anonymous — Tue, 18 Nov 2014 07:41:48 GMT

Hi Kalyan,
Ideally, the server creds are fed into the config file by respective teams, already (& specifically) privy to the info.
Even if YOU are required to provide the passwords for all of the environment, you can simply type them in the config file (no need to encrypt it) & the Talend job will read it. BTW, the 'encryption' which you are talking about is not actually encryption. Its a setting, which only hides the passwords (& not actually encrypt it)
Anyways, you would not want to store environment passwords in your job, instead for them to be furnished to the job at runtime (from different sources would be even better)
MathurM

Re: [resolved] Configuring job to run on different environments

satyavar — Wed, 26 Nov 2014 12:13:32 GMT

Hi
I am reading the values from a csv file and loading the context variables. When i am trying to override the context variable values set by metadata, they are not reacting. Please see the attached screenshot. The password is set to TALEND_2 in Load_Context_Variables job and when it comes to tJavaRow - It is set back to TALEND_1 because of ds_last_load_id object.
Any idea on how can i stop overriding the context variables and only let the values passed from csv?
Thanks
Kalyan

Re: [resolved] Configuring job to run on different environments

Anonymous — Wed, 26 Nov 2014 13:02:45 GMT

hi,
but there's any csv file to tContextLoad in your screenShoot (after tprejob)
https://help.talend.com/search/all?query=tContextLoad

regards
laurent

Re: [resolved] Configuring job to run on different environments

satyavar — Thu, 27 Nov 2014 03:59:30 GMT

Hi
Load_CONTEXT_Variables job in the screenshot is reading the values from a csv file. Its a separate job, which i am planning to apply to all the jobs.
Thanks
Kalyan

Re: [resolved] Configuring job to run on different environments

Anonymous — Thu, 27 Nov 2014 10:13:40 GMT

I am reading the values from a csv file and loading the context variables

but you're not
tFileinputDelimited component to tContextLoad is missing.
regards

Re: [resolved] Configuring job to run on different environments

satyavar — Thu, 27 Nov 2014 10:16:37 GMT

Please see the below screen shot, I am reading the values from csv and using tContextLoad.