topic Re: System.getenv() when run Job from TAC in Talend Studio

System.getenv() when run Job from TAC

vapukov — Sat, 16 Nov 2024 08:09:25 GMT

Hi,

with community edition we are use System.getenv() for retrieve OS variables, it work fine everywhere:

- windows, linux, macOS

- from Studio

- from shell

also same function work under Talend ESB Runtime as expected

but when Job implemented on TAC, this function stop work and return null (in Studio for subscription version - all fine)

variables defined in:

/etc/environment (normally for Community and Runtime)

.bachrc (duplicated when try to resolve)

setenv.sh (duplicated when try to resolve)

server of course was restarted after any changes

look like TAC start processes without passing environment, any recommendations how to fix this would be pretty appreciated !

Thanks, Vlad

Re: System.getenv() when run Job from TAC

Anonymous — Tue, 05 Jun 2018 10:40:48 GMT

I suspect this is because your job servers are not located on your TAC machine and/or you do not have the system environment variables set up on all of your job servers. The jobs do not take any environment settings from the TAC since they do not run on the TAC (unless you have a job server on there as well). When you use a virtual job server (made up of several job servers) you will need to have these variables set up on ALL job servers or the jobs will only work intermittently.

Re: System.getenv() when run Job from TAC

vapukov — Tue, 05 Jun 2018 11:05:18 GMT

it is single server installation - all on same server only SVN on remote

Re: System.getenv() when run Job from TAC

Anonymous — Tue, 05 Jun 2018 11:19:49 GMT

Then this is likely to do with permissions and OS environment variable settings. Which user is running your services (Talend Job Services, etc) and are the environment variables configured for them? What you are trying to do certainly works. I use this all of the time. However it can get frustrating trying to track down what is wrong sometimes. Can you echo the variables from the command line on the machine using the user which runs the Job server service. That is the first thing to try.

Re: System.getenv() when run Job from TAC

vapukov — Tue, 05 Jun 2018 11:23:25 GMT

- /etc/environment - for all users

- talenduser - /home/talenduser/.bashrc

this user from which Talend start

same user used for run ESB Runtime with only /etc/environment settings - and function work from OSGI jobs

P.S.

of course echo from Talenduser show correct variables, it was first what tested

Re: System.getenv() when run Job from TAC

Anonymous — Tue, 05 Jun 2018 13:12:48 GMT

This is strange.

"look like TAC start processes without passing environment, any recommendations how to fix this would be pretty appreciated !" - This is not the case as environment variables are never passed to a Talend job. Using the code you have demonstrated shows you are requesting them from within an already started job. This is an issue with the Java and/or (more likely) your server config. TAC has absolutely no effect on this. It literally just passes the executable to the appropriate jobserver and starts the job. That is all. I suspect your Jobserver service is not running under the user you suspect it is. You can test this by creating a job which uses a tSystem control to try your echo statements.

Re: System.getenv() when run Job from TAC

fdenis — Tue, 05 Jun 2018 13:37:04 GMT

for me, environment variables are load on logon.

TAC is used to Schedule background tasks on multiples runtime servers. you did not need to be logged to run task on Talend runtime environment. Local service have to run but independent sub processes did not need any environment variables.

Re: System.getenv() when run Job from TAC

fdenis — Tue, 05 Jun 2018 13:40:36 GMT

With tac the best way is to use context variables.

Re: System.getenv() when run Job from TAC

Anonymous — Tue, 05 Jun 2018 14:36:20 GMT

I use (and have done for a few years, across versions from 5 forward) a procedure where my implicit context load (for DI jobs) retrieves context variables from a database. So that there is absolutely no difference required from first DEV compilation, through QA to PROD, I configure each of my environments with context database credentials using system variables. This means I can configure each of my environments on creation and do not have to touch my jobs again no matter which environment they run on. I only use the Default context because having multiple contexts will end up with issues no matter how thorough you think you....especially if there is a team of developers. I do this with my ESB work as well, but use PropertyPlaceholder to receive my "context"" variables so that I have exactly the same values across the board. I also add another layer of complexity in many environments as many of my customers do not like human readable values to be stored even in environment settings. For this I use JASYPT. I do not experience this issue unless service users are incorrect OR there is a new Job Server that has not been configured. I have used this on Windows and Unix and apart from the configuration of the servers, the implementation is exactly the same.

I use this routine code in my Implicit Context settings (they can run code as well as accept hardcoded values) and within my jobs sometimes....although I tend to load everything from the Implicit Context if it is suitable.

    /**
     * getEnvironmentVariable: used to retrieve Environment Variables
     * 
     * {talendTypes} String
     * 
     * {Category} Implicit Context Load
     * 
     * {param} string("TalendContextPassword") variableName: the parameter name to be returned

     * 
     * {example} getEnvironmentVariable("TalendContextPassword") # returns "My Password"
     */
	public static String getEnvironmentVariable(String variableName) {
		String returnVal = System.getenv(variableName);

		if (returnVal == null) {
			System.out.println(variableName
					+ " does not exist or holds no value");
		}

		return returnVal;
	}

This works for running on my DEV Studio (my machine is configured with environment variables on Windows), running jobs from the command line (Windows or Linux) and running by starting on TAC. To add even more of a potential for problems to this mix, I automatically load tasks into the TAC using my ESB runtime (when a file is dropped in a folder and the required job is not yet live) by using the MetaServlet API to find the required job in my Nexus and load to the TAC. No contexts are set up in the TAC because this process renders doing that unnecessary. It also enables me to have new jobs that supersede already existing jobs to be used as soon as they are logged in my background database as being the "live" job.

This all works without fault and n my current project I have had no problems with this across 3 environments for the last 6 months. As I said, there is nothing in Talend stopping this from working. It is an implementation problem.

Re: System.getenv() when run Job from TAC

vapukov — Tue, 05 Jun 2018 21:32:42 GMT

@rhall wrote:

This is strange.

TAC has absolutely no effect on this. It literally just passes the executable to the appropriate jobserver and starts the job. That is all. I suspect your Jobserver service is not running under the user you suspect it is. You can test this by creating a job which uses a tSystem control to try your echo statements.

thank You, sometime brain storm help (not in this case :-))) )

just for close all ideas about users

file /etc/environment - affected for all users

https://stackoverflow.com/questions/13046624/how-to-permanently-export-a-variable-in-linux

Re: System.getenv() when run Job from TAC

vapukov — Tue, 05 Jun 2018 21:35:07 GMT

we use of course

but question about - how to read OS variables

Re: System.getenv() when run Job from TAC

Anonymous — Tue, 05 Jun 2018 21:45:31 GMT

Does your job work on the TAC if you manually run it via the command line on that machine? For example, compile on Studio and take the zip, explode it and run it using the script file provided. Is that able to retrieve the system environment variables?

Re: System.getenv() when run Job from TAC

vapukov — Tue, 05 Jun 2018 21:47:11 GMT

@rhall wrote:

I use this routine code in my Implicit Context settings (they can run code as well as accept hardcoded values) and within my jobs sometimes....although I tend to load everything from the Implicit Context if it is suitable.
    /**
     * getEnvironmentVariable: used to retrieve Environment Variables
     * 
     * {talendTypes} String
     * 
     * {Category} Implicit Context Load
     * 
     * {param} string("TalendContextPassword") variableName: the parameter name to be returned

     * 
     * {example} getEnvironmentVariable("TalendContextPassword") # returns "My Password"
     */
	public static String getEnvironmentVariable(String variableName) {
		String returnVal = System.getenv(variableName);

		if (returnVal == null) {
			System.out.println(variableName
					+ " does not exist or holds no value");
		}

		return returnVal;
	}
This all works without fault and n my current project I have had no problems with this across 3 environments for the last 6 months. As I said, there is nothing in Talend stopping this from working. It is an implementation problem.

I use similar construction (little different, but still - it work last 2.5 year) ... until this case

Tell me please - what implementation problem it could be? what could restrict read settings available for 100% users on this server?

Re: System.getenv() when run Job from TAC

vapukov — Tue, 05 Jun 2018 21:49:50 GMT

@rhall wrote:

Does your job work on the TAC if you manually run it via the command line on that machine? For example, compile on Studio and take the zip, explode it and run it using the script file provided. Is that able to retrieve the system environment variables?

thank You, will test and this also

Re: System.getenv() when run Job from TAC

vapukov — Tue, 05 Jun 2018 22:01:04 GMT

@rhall wrote:

Does your job work on the TAC if you manually run it via the command line on that machine? For example, compile on Studio and take the zip, explode it and run it using the script file provided. Is that able to retrieve the system environment variables?

all work from ssh - as it must

and not correct from TAC:

and again correct from karaf:

Re: System.getenv() when run Job from TAC

Anonymous — Tue, 05 Jun 2018 22:18:13 GMT

Did you say that your TAC, Job Server and Runtime are all running on the same server?

Also, where is it getting the context_filepath from? That is the same in all 3 tests.

Re: System.getenv() when run Job from TAC

vapukov — Tue, 05 Jun 2018 22:22:45 GMT

@rhall wrote:

Did you say that your TAC, Job Server and Runtime are all running on the same server?

Also, where is it getting the context_filepath from? That is the same in all 3 tests.

yes - all 3 on same server

yes and no - this is different case (and as You can see - on TAC - variable is NULL, not the same)

on each server and/or machine defined 2 variables:

- talend_env - dev/prod/local

- talend_context_path - base folder for talenduser config files, it could be not only context file, it could be templates for emails, some other files and etc

if variables defined, we have combined name for current config file:

/opt/talenduser/config/context_prod.csv
/opt/talenduser/config/context_local.csv
/opt/talenduser/config/context_dev.csv

if variables - not defined (or not accessible) - default context file:

/opt/talenduser/config/context_dev.csv

Re: System.getenv() when run Job from TAC

Anonymous — Tue, 05 Jun 2018 22:42:09 GMT

Can you add a call to retrieve the PATH variable in your job? This should exist for all your users. My suspicion is that this will be returned. If that is the case it suggests that the user running the job (or the Job server) does not have access to those variables. Sorry to go back to this, but that can be the only reason for this. The Java that works in the ESB is exactly the same (I assume) as the Java that is running in the job in the job server and on the Studio. All the Job Server does is start the job. The job runs in its own JVM as essentially a standalone application. The only thing that can possibly cause this is a permissions/user issue. It is incredibly easy to have different users running Talend services, particularly on Linux.

Re: System.getenv() when run Job from TAC

vapukov — Tue, 05 Jun 2018 23:33:17 GMT

PATH is work

and much more interesting - ~~it work definitely from /etc/environment:~~ (edited) ... it strange

root@talend-prod-02:/opt/test/testEnvVariables# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/opt/TalendTools-6.4.1/mongodb/bin
root@talend-prod-02:/opt/test/testEnvVariables# su talenduser
talenduser@talend-prod-02:/opt/test/testEnvVariables$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
talenduser@talend-prod-02:/opt/test/testEnvVariables$ cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
talend_env="prod"
talend_context_path="/opt/talenduser/config/"

so, ~~it read /etc/environment~~ ... it take it from ...

PATH from /etc/environment:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
PATH under talenduser shell:

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
PATH from TAC
/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games|

Re: System.getenv() when run Job from TAC

vapukov — Wed, 06 Jun 2018 00:06:26 GMT

AND TAC - do not show HOME

----------+---=|
|talend_env|talend_context_path|context_filepath                      |system_path                                             |home|
|=---------+-------------------+--------------------------------------+--------------------------------------------------------+---=|
|null      |null               |/opt/talenduser/config/context_dev.csv|/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games|null|

it must be:

|talend_env|talend_context_path    |context_filepath                        |system_path                  |home           |
|=---------+-----------------------+----------------------------------------+-----------------------------+--------------=|
|local     |/opt/talenduser/config/|/opt/talenduser/config/context_local.csv|/usr/bin:/bin:/usr/sbin:/sbin|/Users/vladimir|
'----------+-----------------------+----------------------------------------+-----------------------------+---------------'

(samples for local)

Warning: Parameter "dwh_export_path" is a new parameter of Implicit_Context_Context
.----------+-----------------------+---------------------------------------+----------------------------------------------------------------------------------------+----------------.
|                                                                                     tLogRow_1                                                                                      |
|=---------+-----------------------+---------------------------------------+----------------------------------------------------------------------------------------+---------------=|
|talend_env|talend_context_path    |context_filepath                       |system_path                                                                             |home            |
|=---------+-----------------------+---------------------------------------+----------------------------------------------------------------------------------------+---------------=|
|prod      |/opt/talenduser/config/|/opt/talenduser/config/context_prod.csv|/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games|/home/talenduser|
'----------+-----------------------+---------------------------------------+----------------------------------------------------------------------------------------+----------------'