One Star

Derived Context Variables

Hi Team,

 

Lets say I have a context variables var1 and var2.

 

var1="abc"

I need var2 to be var1+" def"

 

Both var1 and var2 are on the same context group. How can I achieve that?

 

Tags (2)
19 REPLIES
Twelve Stars TRF
Twelve Stars

Re: Derived Context Variables

Hi,
Just write context.var2 = context.var1 + " def" in a tJavaRow (or tJavaXxx) component.

TRF
One Star

Re: Derived Context Variables

Hi TRF,

I needed to use it inside the context group itself and assign to another
variable!
Is that possible?
Twelve Stars TRF
Twelve Stars

Re: Derived Context Variables

No, you cannot have code in the place where context variable are defined. Dynamic values must come from tJavaXxx components. You may also define global variables in a PreJob wich is a good practice to achieve initialization tasks when job starts.

TRF
One Star

Re: Derived Context Variables

Thanks TRF.
To me it sounds inefficient. Just because Talend allows only literal values
over there, we have to do the overhead of writing all those prejob codes.
Twelve Stars TRF
Twelve Stars

Re: Derived Context Variables

You have no choice

TRF
Twelve Stars

Re: Derived Context Variables

I don't understand what is inefficient about this. Maybe I am misunderstanding? The only limitation regarding this issue is that you cannot add code to be performed by the Context variables themselves, however you can write a job that can be used in ALL of your jobs to carry out this code. If you are happy writing Java you can perform this sort of functionality with Java Routines or a tJavaXXX (suggested by @TRF) as well. You can also store the context variables in a database and perform any config calculations on this data during import before the jobs start.

If you can elaborate on why this is problematic I believe that myself or @TRF can point you towards an elegant solution to your problem. 

Rilhia Solutions
Ten Stars

Re: Derived Context Variables


rhall_2_0 wrote:

I don't understand what is inefficient about this. Maybe I am misunderstanding?... 


It normal situation, when anybody try to implement logic in new tools exactly as it was somewhere else, or exactly as it designed

Similar questions regular like - "we do the same in Informatica (Mulesoft, ODI, Pentaho ... ) and want exactly this"

 

It wrong ways of course.

Unfortunately - not possible to give full answer without understanding expected logic, there are many ways (including suggested by You)

 

for example source idea:

var1="abc"
I need var2 to be var1+" def"

could be transformed to:

baseVar1="abc" in context group

and on feature steps in Job:
var2=baseVar+" def"

 

-----------
Six Stars

Re: Derived Context Variables

Hi @rhall_2_0

 

Thanks for your reply. Actually no it is not problematic and Thanks @TRF and Yourself for the suggested solutions. Smiley Happy

 

Why I say it is inefficient is, we are setting up parameters on the context groups and therefore would expect them to be self contained and not require some other prejobs to complete their interpretation. What we are having to work with is workarounds, which I am happy to do if there are no other alternatives but should be at-least raised for future releases.

 

Coming to the ask, I would want to compute my data directories according to a particular flag that helps me have different working areas on the same server for parallel testing.

 

For e.g. /a/b/c/uat1/data and /a/b/c/uat2/data

 

Now the approaches I am able to think off are (Am Open to suggestions) :

1.) Different Context Groups for each uat1/uat2 -- I am not too keen on this, because for 5-6 parameters I will have to re-do all 300 odd parameters. More headache for the developer(unless there are any sleek shortcuts to do it) and also bulkier builds!

2.) Using Prejob with tJava components -- I might have to go with this approach. But the problem I can think off is, even for parameter interpretation, all the code is not in one place. We have our contexts that create placeholders and then again another job that actually interprets values, which makes it less than desirable for me.

 

P.S: I have switched to a different username, but still the same user.

 

Thanks 

Rabi Panda

Ten Stars

Re: Derived Context Variables

for this I use only 1 context variable! :-)

 

it name - context.env (I use 3 primary context group, but no limits at all)

prod - "prod"

qual - "qual"

dev - "dev"

 

in all other components (where it necessary ) all names concat like:

source_path == context.sourcefolder+"_"+context.env+"/data"

target_path == context.targetfolder+"_"+context.env+"/data"

 

and etc

same context.env used and run SubJob with dynamic context

 

 

When I need more than 3 - I just initialise context.env with individual value

 

Plus this variables used for load context values from csv file as part of filename

context.contextfilepath+"context_"+context.env+".csv"

and in any Job only 2 variables have value

- context.contextfilepath

- context.env

 

it give extremely flexibility

-----------
Six Stars

Re: Derived Context Variables


Hi @vapukov

 

Similar questions regular like - "we do the same in Informatica (Mulesoft, ODI, Pentaho ... ) and want exactly this"

 

It wrong ways of course.

 

Thanks for your reply. 
Considering Talend is relatively new in the ETL space comparison is inevitable.
We could argue that comparing is wrong, I am of the opinion if something works for one should be scope for improvement for the other.

 

Regards,

Rabi Panda

 

Six Stars

Re: Derived Context Variables


vapukov wrote:

for this I use only 1 context variable! :-)

 

it name - context.env (I use 3 primary context group, but no limits at all)

prod - "prod"

qual - "qual"

dev - "dev"

 

in all other components (where it necessary ) all names concat like:

source_path == context.sourcefolder+"_"+context.env+"/data"

target_path == context.targetfolder+"_"+context.env+"/data"

 

and etc

same context.env used and run SubJob with dynamic context

 

 

When I need more than 3 - I just initialise context.env with individual value

 

Plus this variables used for load context values from csv file as part of filename

context.contextfilepath+"context_"+context.env+".csv"

and in any Job only 2 variables have value

- context.contextfilepath

- context.env

 

it give extremely flexibility


Yup we could do this, but then again I am sure you would agree, defining/computing the above expression at one place using it at the rest sounds a better approach.

 

Regards,

Rabi Panda

Twelve Stars

Re: Derived Context Variables

First of all, never use Contexts other than default. Many people confuse the Context terminology. A context is your set of context variables (DEV, TEST, PROD, default, etc). A Context Group is a batch of related context variables. A Context variables is....well you know that. Using multiple Contexts just adds risk to process. You can mitigate this if you want to have multiple environments by using flat files or a database with the Implicit Context Load. You can make the Implicit Context Load as simple or as complicated (and secure, tailorable, etc) as you like. It is briefly described here https://help.talend.com/display/TalendOpenStudioforDataIntegrationUserGuide60EN/C.3.4+Configuring+th...

Rilhia Solutions
Ten Stars

Re: Derived Context Variables

may be not for 100% agree with - never/ever

 

life is different, so in some cases - it useful

 

in case when each job has it own context file - it easy to change file, but in case when many jobs share connection and other information ... better to have 

TAC allow change single variable for single Job, with Community version it not true, so You will need or add command line parameter for point to different context file (database). 

 

other case - my client work on windows server, I'm on Mac - file path will be different, so we have 2 Context at least, 1 point to my local context file, other point to his

 

 

on different projects - meet with all possible variant, but not sure which better

-----------
Twelve Stars

Re: Derived Context Variables

Many of those issues (file paths, for example) can be resolved with operating system Environment Variables. My current use of Context Variables only uses default, makes use of environment variables and encrypted db parameters and Java routines in the Implicit Context Load config (yes, routines work there) allowing a pretty secure but easily malleable Context system. My Prod and Test environments cannot be changed without access to the context db, while the developers can all share context settings or set their own. This is pretty complicated, but once configured all it requires is administration of the context db.

The reason I dislike Contexs (Prod, Test, Dev, etc) is that if you have child jobs being called by parent jobs (who doesn't) you can easily run into issues when the Contexts are set incorrectly. This doesn't need to even be thought about with the system I use.

Rilhia Solutions
Ten Stars

Re: Derived Context Variables

agree, Your way is good.

 

I also have experience with OS variables, working solution.

 

for now I choose as flexible for my needs, next way:

in tRunJob:

Screen Shot 2017-05-05 at 10.50.37 PM.png

and in context:

Screen Shot 2017-05-05 at 10.53.03 PM.png

 

 

 

 

 

-----------
Six Stars

Re: Derived Context Variables

Thanks @rhall_2_0

 

Trying to implement that.

However I am not able to reference the variables set using ImplicitCOntextLoad the same way we reference other contexts.

i.e. context.var1

 

Can you please help!

Twelve Stars

Re: Derived Context Variables

You have to set the context variables up in the Job. Have you done this? By that I mean simply either add your Context Group (if you have them grouped) or create each variable in the contexts section of your job. Once they exist, the implicit context load will bring them in.

Rilhia Solutions
Six Stars

Re: Derived Context Variables

In fact ignore the query, I had not defined it anywhere. Works for me now. Thanks for your help

Twelve Stars

Re: Derived Context Variables

No problem

Rilhia Solutions