One Star

Dynamically define Job Contexts

I want to create a Reference job that can be auto-populated with new or updated job contexts.

A typical job context can be something like "DEV, "UAT", "PROD". But let's say down the road I want to add another job context called "PREPROD".

 

My goal is to simply add this new job context to a database table, and when the job loads, the new job context becomes visible.

 

Can this be done, and if so... how?

 

 

  • Data Integration
3 REPLIES
Nine Stars

Re: Dynamically define Job Contexts

Yes it can be done.

Contexts in Talend are not very nice to use. At design time, we may be aware of DEV, TEST and PROD, but later on these environments may grow beyond that. To have to recompile all jobs to introduce new environments is just a pain. So (as you suggested) we can use a database to hold context variables. 

 

The way that I do this is to use OS environment variables. Your jobs will run on several environments. There will be your developer (Studio) environments, your TEST and your PROD environments. Each of these environments can have OS environment variables configured for your context variable db settings and your environment (DEV, TEST, PROD ....individual developer environments). These OS environment variables can be retrieved in Java using the following code....

System.getenv(variableName)

Now the implicit context load functionality can be used here. The variables that you have to specify do not have to be hardcoded with String values. You can use Java here too. So what you can do is configure your implicit content load settings to use the code above, pointing to the OS environment settings for the machine you are working on (the code is running on). 

Then all you have to do is maintain your context variable database(s). I find this works much better than using the fixed Contexts provided by Talend. Simply stick to the "Default" context and supply your variables at runtime using the above method.

Rilhia Solutions
One Star

Re: Dynamically define Job Contexts

Thanks for the quick reply, Richard.

 

While the proposed solution would work, there are a few gotchas:

1. We're running Talend in the cloud, so an OS-level context variable I don't think would apply here. (Nor would it be accessible? Unless I'm mis-understanding something.. I'm a total noob to Talend)

 

2.If we were running all of our jobs locally at the server level, some of our servers are clustered, running multiple instances. Setting an OS level environment variable can only be done once, even if there are instances running multiple environments. (eg: DEV / UAT)

 

As I'm typing this I'm wondering if it's a better idea to create a blank reference job with a given set of job contexts, then, create joblets as connectors to various platforms: ie: a joblet to connect to Salesforce, another to Netsuite, etc...The joblets will be tasked with getting context key/value data based on the current job context, with the parent reference job inheriting the joblet context vars. Interested to know your thoughts on this approach.

Having a fair amount of experience in SSIS I am acutely familiar with the pain of having to go back and re-factor hundreds of packages because of poor design/implementation so naturally I want to avoid that situation. :-)


Nine Stars

Re: Dynamically define Job Contexts

If you cannot make use of OS environment settings, an easier way of working with this is to place a flat file in the same location on all machines running Talend. Read the DB credentials from this file (maybe using some Java) and then pass the db credentials derived from that file to the implicit context variables settings. This can all be done using a Talend Routine. If you are concerned about storing passwords in flat files you can make use of jasypt which is supported by Talend.

Rilhia Solutions