Twelve Stars

System.getenv() when run Job from TAC

Hi,

 

with community edition we are use System.getenv() for retrieve OS variables, it work fine everywhere:

- windows, linux, macOS 

 

- from Studio

- from shell

 

also same function work under Talend ESB Runtime as expected

 

but when Job implemented on TAC, this function stop work and return null (in Studio for subscription version - all fine)

 

variables defined in:

/etc/environment (normally for Community and Runtime)

.bachrc (duplicated when try to resolve)

setenv.sh (duplicated when try to resolve)

 

server of course was restarted after any changes

 

look like TAC start processes without passing environment, any recommendations how to fix this would be pretty appreciated !

Thanks, Vlad

-----------
1 ACCEPTED SOLUTION

Accepted Solutions
Twelve Stars

Re: System.getenv() when run Job from TAC

ok, so "partial" solution

"partial" - because I fix the issue, but right now I have not time to check - what exactly step was only needed

 

1. JobServer start as not interactive process (.sh script), and for not interactive processes OS environment do not pass

2. for this reason I add variables into:

  • /etc/environment
  • ~/.bashrc
  • jobserver/agent/start_rs.sh
  • /etc/bash.bashrc
  • ~/.bash_profile

this is not proper way, just because I have not time - I add variables in any place where it could be taken. proper do it 1 by 1 and reboot after each change for check, but I was very limited in time.

 

as result, all work as expected:

.----------+-----------------------+---------------------------------------.
|                                tLogRow_1                                 |
|=---------+-----------------------+--------------------------------------=|
|talend_env|talend_context_path    |context_filepath                       |
|=---------+-----------------------+--------------------------------------=|
|prod      |/opt/talenduser/config/|/opt/talenduser/config/context_prod.csv|
'----------+-----------------------+---------------------------------------'
-----------
25 REPLIES
Fifteen Stars

Re: System.getenv() when run Job from TAC

I suspect this is because your job servers are not located on your TAC machine and/or you do not have the system environment variables set up on all of your job servers. The jobs do not take any environment settings from the TAC since they do not run on the TAC (unless you have a job server on there as well). When you use a virtual job server (made up of several job servers) you will need to have these variables set up on ALL job servers or the jobs will only work intermittently.

Rilhia Solutions
Twelve Stars

Re: System.getenv() when run Job from TAC

it is single server installation - all on same server Smiley Happy only SVN on remote

-----------
Fifteen Stars

Re: System.getenv() when run Job from TAC

Then this is likely to do with permissions and OS environment variable settings. Which user is running your services (Talend Job Services, etc) and are the environment variables configured for them? What you are trying to do certainly works. I use this all of the time. However it can get frustrating trying to track down what is wrong sometimes. Can you echo the variables from the command line on the machine using the user which runs the Job server service. That is the first thing to try.

Rilhia Solutions
Twelve Stars

Re: System.getenv() when run Job from TAC

- /etc/environment - for all users

- talenduser - /home/talenduser/.bashrc

this user from which Talend start

 

same user used for run ESB Runtime with only /etc/environment settings - and function work from OSGI jobs

 

P.S.

of course echo from Talenduser show correct variables, it was first what tested

-----------
Fifteen Stars

Re: System.getenv() when run Job from TAC

This is strange.

 

"look like TAC start processes without passing environment, any recommendations how to fix this would be pretty appreciated !" -  This is not the case as environment variables are never passed to a Talend job. Using the code you have demonstrated shows you are requesting them from within an already started job. This is an issue with the Java and/or (more likely) your server config. TAC has absolutely no effect on this. It literally just passes the executable to the appropriate jobserver and starts the job. That is all. I suspect your Jobserver service is not running under the user you suspect it is. You can test this by creating a job which uses a tSystem control to try your echo statements.

 

Rilhia Solutions
Ten Stars

Re: System.getenv() when run Job from TAC

for me, environment variables are load on logon.

TAC is used to Schedule background tasks on multiples runtime servers. you did not need to be logged to run task on Talend runtime environment. Local service have to run but independent sub processes did not need any environment variables.

Francois Denis

Don't forget to tag when it's "solved"!

Ten Stars

Re: System.getenv() when run Job from TAC

With tac the best way is to use context variables.

Francois Denis

Don't forget to tag when it's "solved"!

Fifteen Stars

Re: System.getenv() when run Job from TAC

I use (and have done for a few years, across versions from 5 forward) a procedure where my implicit context load (for DI jobs) retrieves context variables from a database. So that there is absolutely no difference required from first DEV compilation, through QA to PROD, I configure each of my environments with context database credentials using system variables. This means I can configure each of my environments on creation and do not have to touch my jobs again no matter which environment they run on. I only use the Default context because having multiple contexts will end up with issues no matter how thorough you think you....especially if there is a team of developers. I do this with my ESB work as well, but use PropertyPlaceholder to receive my "context"" variables so that I have exactly the same values across the board. I also add another layer of complexity in many environments as many of my customers do not like human readable values to be stored even in environment settings. For this I use JASYPT. I do not experience this issue unless service users are incorrect OR there is a new Job Server that has not been configured. I have used this on Windows and Unix and apart from the configuration of the servers, the implementation is exactly the same.

 

I use this routine code in my Implicit Context settings (they can run code as well as accept hardcoded values) and within my jobs sometimes....although I tend to load everything from the Implicit Context if it is suitable.

 

    /**
     * getEnvironmentVariable: used to retrieve Environment Variables
     * 
     * {talendTypes} String
     * 
     * {Category} Implicit Context Load
     * 
     * {param} string("TalendContextPassword") variableName: the parameter name to be returned

     * 
     * {example} getEnvironmentVariable("TalendContextPassword") # returns "My Password"
     */
	public static String getEnvironmentVariable(String variableName) {
		String returnVal = System.getenv(variableName);

		if (returnVal == null) {
			System.out.println(variableName
					+ " does not exist or holds no value");
		}

		return returnVal;
	}

This works for running on my DEV Studio (my machine is configured with environment variables on Windows), running jobs from the command line (Windows or Linux) and running by starting on TAC. To add even more of a potential for problems to this mix, I automatically load tasks into the TAC using my ESB runtime (when a file is dropped in a folder and the required job is not yet live) by using the MetaServlet API to find the required job in my Nexus and load to the TAC. No contexts are set up in the TAC because this process renders doing that unnecessary. It also enables me to have new jobs that supersede already existing jobs to be used as soon as they are logged in my background database as being the "live" job.

 

This all works without fault and n my current project I have had no problems with this across 3 environments for the last 6 months. As I said, there is nothing in Talend stopping this from working. It is an implementation problem.

 

 

Rilhia Solutions
Twelve Stars

Re: System.getenv() when run Job from TAC


@rhall_2_0 wrote:

This is strange.

 

TAC has absolutely no effect on this. It literally just passes the executable to the appropriate jobserver and starts the job. That is all. I suspect your Jobserver service is not running under the user you suspect it is. You can test this by creating a job which uses a tSystem control to try your echo statements.

 


thank You, sometime brain storm help Smiley Happy (not in this case :-))) )

 

just for close all ideas about users

 

file /etc/environment - affected for all users 

https://stackoverflow.com/questions/13046624/how-to-permanently-export-a-variable-in-linux


 

 

-----------
Twelve Stars

Re: System.getenv() when run Job from TAC

we use of course

but question about - how to read OS variables Smiley Happy

-----------
Fifteen Stars

Re: System.getenv() when run Job from TAC

Does your job work on the TAC if you manually run it via the command line on that machine? For example, compile on Studio and take the zip, explode it and run it using the script file provided. Is that able to retrieve the system environment variables?

Rilhia Solutions
Twelve Stars

Re: System.getenv() when run Job from TAC


@rhall_2_0 wrote:

 

I use this routine code in my Implicit Context settings (they can run code as well as accept hardcoded values) and within my jobs sometimes....although I tend to load everything from the Implicit Context if it is suitable.

 

    /**
     * getEnvironmentVariable: used to retrieve Environment Variables
     * 
     * {talendTypes} String
     * 
     * {Category} Implicit Context Load
     * 
     * {param} string("TalendContextPassword") variableName: the parameter name to be returned

     * 
     * {example} getEnvironmentVariable("TalendContextPassword") # returns "My Password"
     */
	public static String getEnvironmentVariable(String variableName) {
		String returnVal = System.getenv(variableName);

		if (returnVal == null) {
			System.out.println(variableName
					+ " does not exist or holds no value");
		}

		return returnVal;
	}

 

This all works without fault and n my current project I have had no problems with this across 3 environments for the last 6 months. As I said, there is nothing in Talend stopping this from working. It is an implementation problem.

 

 


I use similar construction (little different, but still - it work last 2.5 year) ... until this case

 

Tell me please - what implementation problem it could be? Smiley Wink what could restrict read settings available for 100% users on this server?

Screen Shot 2018-06-06 at 08.45.54.pngScreen Shot 2018-06-06 at 08.18.43.png

-----------
Twelve Stars

Re: System.getenv() when run Job from TAC


@rhall_2_0 wrote:

Does your job work on the TAC if you manually run it via the command line on that machine? For example, compile on Studio and take the zip, explode it and run it using the script file provided. Is that able to retrieve the system environment variables?


thank You, will test and this also

-----------
Twelve Stars

Re: System.getenv() when run Job from TAC


@rhall_2_0 wrote:

Does your job work on the TAC if you manually run it via the command line on that machine? For example, compile on Studio and take the zip, explode it and run it using the script file provided. Is that able to retrieve the system environment variables?


Screen Shot 2018-06-06 at 09.00.01.png

 

all work from ssh - as it must Smiley Happy

 

and not correct from TAC:

 

Screen Shot 2018-06-06 at 09.04.00.png

 

and again correct from karaf:

Screen Shot 2018-06-06 at 09.05.31.png

 

-----------
Fifteen Stars

Re: System.getenv() when run Job from TAC

Did you say that your TAC, Job Server and Runtime are all running on the same server? 

 

Also, where is it getting the context_filepath from? That is the same in all 3 tests.

Rilhia Solutions
Twelve Stars

Re: System.getenv() when run Job from TAC


@rhall_2_0 wrote:

Did you say that your TAC, Job Server and Runtime are all running on the same server? 

 

Also, where is it getting the context_filepath from? That is the same in all 3 tests.


yes - all 3 on same server

yes and no - this is different case Smiley Happy (and as You can see - on TAC - variable is NULL, not the same)

 

on each server and/or machine defined 2 variables:

- talend_env - dev/prod/local

- talend_context_path - base folder for talenduser config files, it could be not only context file, it could be templates for emails, some other files and etc

 

if variables defined, we have combined name for current config file:

  • /opt/talenduser/config/context_prod.csv
  • /opt/talenduser/config/context_local.csv
  • /opt/talenduser/config/context_dev.csv

if variables - not defined (or not accessible) - default context file:

  • /opt/talenduser/config/context_dev.csv

 

 

-----------
Fifteen Stars

Re: System.getenv() when run Job from TAC

Can you add a call to retrieve the PATH variable in your job? This should exist for all your users. My suspicion is that this will be returned. If that is the case it suggests that the user running the job (or the Job server) does not have access to those variables. Sorry to go back to this, but that can be the only reason for this. The Java that works in the ESB is exactly the same (I assume) as the Java that is running in the job in the job server and on the Studio. All the Job Server does is start the job. The job runs in its own JVM as essentially a standalone application. The only thing that can possibly cause this is a permissions/user issue. It is incredibly easy to have different users running Talend services, particularly on Linux.

Rilhia Solutions
Twelve Stars

Re: System.getenv() when run Job from TAC

PATH is work

and much more interesting - it work definitely from /etc/environment: (edited) ... it strange

root@talend-prod-02:/opt/test/testEnvVariables# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/opt/TalendTools-6.4.1/mongodb/bin
root@talend-prod-02:/opt/test/testEnvVariables# su talenduser
talenduser@talend-prod-02:/opt/test/testEnvVariables$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
talenduser@talend-prod-02:/opt/test/testEnvVariables$ cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
talend_env="prod"
talend_context_path="/opt/talenduser/config/"

so, it read /etc/environment ... it take it from ...

Screen Shot 2018-06-06 at 10.32.40.png

 

PATH from /etc/environment:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
PATH under talenduser shell:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
PATH from TAC
/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games|
-----------
Twelve Stars

Re: System.getenv() when run Job from TAC

AND TAC - do not show HOME Smiley Happy

 

----------+---=|
|talend_env|talend_context_path|context_filepath                      |system_path                                             |home|
|=---------+-------------------+--------------------------------------+--------------------------------------------------------+---=|
|null      |null               |/opt/talenduser/config/context_dev.csv|/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games|null|


it must be:

|talend_env|talend_context_path    |context_filepath                        |system_path                  |home           |
|=---------+-----------------------+----------------------------------------+-----------------------------+--------------=|
|local     |/opt/talenduser/config/|/opt/talenduser/config/context_local.csv|/usr/bin:/bin:/usr/sbin:/sbin|/Users/vladimir|
'----------+-----------------------+----------------------------------------+-----------------------------+---------------'

 (samples for local) 

 

Warning: Parameter "dwh_export_path" is a new parameter of Implicit_Context_Context
.----------+-----------------------+---------------------------------------+----------------------------------------------------------------------------------------+----------------.
|                                                                                     tLogRow_1                                                                                      |
|=---------+-----------------------+---------------------------------------+----------------------------------------------------------------------------------------+---------------=|
|talend_env|talend_context_path    |context_filepath                       |system_path                                                                             |home            |
|=---------+-----------------------+---------------------------------------+----------------------------------------------------------------------------------------+---------------=|
|prod      |/opt/talenduser/config/|/opt/talenduser/config/context_prod.csv|/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games|/home/talenduser|
'----------+-----------------------+---------------------------------------+----------------------------------------------------------------------------------------+----------------'
-----------
Twelve Stars

Re: System.getenv() when run Job from TAC

and one more test, whoami from TAC:

[INFO ]: sugarcrm.testenvvariables_0_1.testEnvVariables - tSystem_1 - Command to execute: 'whoami '.
[INFO ]: sugarcrm.testenvvariables_0_1.testEnvVariables - tSystem_1 - The command has been executed successfully.
talenduser

 

 

little more extend example:

prepare .sh file:

#!/bin/sh

whoami
echo $talend_env
echo $talend_context_path

run from talenduser:

root@talend-prod-02:/opt/test/testEnvVariables# su talenduser
talenduser@talend-prod-02:/opt/test/testEnvVariables$ /opt/test/testEnvVariables/check_env.sh
talenduser
prod
/opt/talenduser/config/

 run from TAC:

[INFO ]: sugarcrm.testenvvariables_0_1.testEnvVariables - tSystem_2 - The command has been executed successfully.
talenduser


[statistics] disconnected

and we have:

- user correct and same

- job run from proper username

- user have permission for file

- but JobServer - do not pass OS variables to this process

 

 

all variables (printenv) from run with JobServer:

UPSTART_INSTANCE=
SHLVL=2
OLDPWD=/opt/TalendTools-6.4.1/jobserver/agent
runlevel=2
UPSTART_JOB=rc
_=./start_rs.sh
TERM=linux
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
RUNLEVEL=2
PREVLEVEL=N
UPSTART_EVENTS=runlevel
TALEND_HOME=/opt/TalendTools-6.4.1
PWD=/opt/TalendTools-6.4.1/jobserver/agent/TalendJobServersFiles/repository/SUGARCRM_testEnvVariables_20180606_125412_CWdfx/testEnvVariables
JAVA_HOME=/usr/lib/jvm/java-8-oracle/jre
previous=N

 

-----------
Ten Stars

Re: System.getenv() when run Job from TAC

it's a good question, but you are using an an ETL software. So you are accepting some abstraction level. Talend is generating Java ,and give you the possibility to add java code into Talend job's. environment variables exist to share common data on a lower execution level. if you want to respect ETL minding think about data not about java,

if you want to have local variable on each server, you can use load local context file.

but you cannot say Talend is not working. Talend is not an java IDE but is using a java based IDE   

Regards

Francois Denis

Don't forget to tag when it's "solved"!

Twelve Stars

Re: System.getenv() when run Job from TAC

@fdenis wrote:

it's a good question, but you are using an an ETL software. So you are accepting some abstraction level. Talend is generating Java ,and give you the possibility to add java code into Talend job's. environment variables exist to share common data on a lower execution level. if you want to respect ETL minding think about data not about java,

if you want to have local variable on each server, you can use load local context file.

but you cannot say Talend is not working. Talend is not an java IDE but is using a java based IDE   

Regards

Thank You

 

for all time working with Talend, I do not remember case when I tell something similar Smiley Happy
 

 

but in any case - I try to find the way, how to achieve what I want?
I don't try to find how to achieve this by other ways  - I now them.


I just use this construction long time, and I have it many projects pre-defined, and I want be sure - this is not possible with TAC,

Plus some other scripts connected to Talend (not my) - also use same or similar variables, and I do not want ask 3rd party developers to change  this without 100% sure - it is no other ways.



What I expect to find:

- or confirmation - this is not work art all, but as You can read above - Richard report he use this construction with TAC, so what difference and what wrong with my TAC? (karaf is also top level java container - but it work with ebb runtime)
- or some tricks - how to achieve this, like may be need edit some JobServer settings for mange him

-----------
Fifteen Stars

Re: System.getenv() when run Job from TAC

I believe this is actually a Linux *feature* rather than a Talend bug after some reading. The quickest piece to look at (for getting on the right track of the problem) can be found here: https://serverfault.com/questions/374404/service-command-and-environment-variables

 

I recalled that I did have an issue with environment variables a long time back, but the sys-admin guys fixed it for me pretty quickly by "sorting the service initialisation" or something to that effect. My assumption was that they had set it up against the wrong user (a situation I have seen on many an occasion on Windows systems). I suspect that the issue was that the environment variables were stripped out by the service command.

 

The reason this does not happen with the Karaf (ESB Runtime) is that this is run as a daemon process and I suspect that is the case for Tomcat (TAC) as well. 

 

Sorry if it sounded like I wasn't paying attention to what you were saying or that I didn't believe you, I was just trying to rule out everything I could think of first. It is kind of hard to resolve these sorts of things without access to "see for yourself". Hopefully this will help you find a workaround. 

 

I'd also like to say there is absolutely nothing wrong with using Java with Talend. In fact I would argue that used correctly, it is one of the biggest pros of using Talend. There is NOTHING I cannot interact with using Talend because of the ability to build in other people's APIs. The trick is to get the usage right and don't reinvent the wheel.....unless the wheel (contexts in this case) is already pretty broken of course :-)

Rilhia Solutions
Twelve Stars

Re: System.getenv() when run Job from TAC

Thank You Richard - no any issues

 

troubleshooting is always pain Smiley Happy and any idea is useful

most important to know and be sure - at least one person have expected result ... that all, its mean solution is somewhere near Smiley Happy just need to find it

I also ask my system admin to check, may be we found what wrong

 

and other case - when just have answer "it is not possible, it not supported" ... this is also answer 

thanks again, regards, Vlad

-----------
Twelve Stars

Re: System.getenv() when run Job from TAC

ok, so "partial" solution

"partial" - because I fix the issue, but right now I have not time to check - what exactly step was only needed

 

1. JobServer start as not interactive process (.sh script), and for not interactive processes OS environment do not pass

2. for this reason I add variables into:

  • /etc/environment
  • ~/.bashrc
  • jobserver/agent/start_rs.sh
  • /etc/bash.bashrc
  • ~/.bash_profile

this is not proper way, just because I have not time - I add variables in any place where it could be taken. proper do it 1 by 1 and reboot after each change for check, but I was very limited in time.

 

as result, all work as expected:

.----------+-----------------------+---------------------------------------.
|                                tLogRow_1                                 |
|=---------+-----------------------+--------------------------------------=|
|talend_env|talend_context_path    |context_filepath                       |
|=---------+-----------------------+--------------------------------------=|
|prod      |/opt/talenduser/config/|/opt/talenduser/config/context_prod.csv|
'----------+-----------------------+---------------------------------------'
-----------