I want to implement a totally dynamic job in talend.
By dynamic, I mean that I would have a file, which will have the metadata -the source, number of columns, their names, targets, their data transformations etc. I am hoping to use the jobscript and parametrise all or most of the components.
1. Can we write if else loops and for loops in the job script?
2. Can the jobscript be used with a context file?
3. Does jobscript give this sort of flexibility or I would be better off writing custom components?
Thanks in advance!
Why not use the Metadata Bridge Excel Import?
You define all the info about the job, even transformations, and then import it, and it generates your job.
It all depends why you want to generate the jobs. Why not have a single job that reads your metadata and do the processing according to the metadata.
Talend is a GUI for graphically designing job. Hence, leverage that, and create a set of generic jobs that read your metadata file and do the correct processing.
To answer your specific questions:
1. You cannot write if else loops section in the jobscript to automate creation of the job. The full jobscript should be already defined. It is a textual definition of your job. Not really a full language.
2. Not sure what you mean by jobscript used with a context file. Please elaborate.
3. Well I gave you 2 different approaches above. It depends on your need.
Thank you for your response!
I figured jobscript would best suite because there would be multiple metadata files and I wanted them to be consumed with a generic job.
I did not know we could do meta bridge import from excel files. I will try that when I get the subscription version.
Why not have a single job that reads your metadata and do the processing according to the metadata.
Could you please elaborate on this?
This actually was my initial plan. Please suggest corrections/improvements wherever you think would be required - If I create a job to read the metadata, It will read the source and its schema and target and its schema. This can be done by tsetdynamicschema. But while reading the rules to be applied to the source, for some rows, what if I have to join a table to get the target value, while in some rows, I will just need to use an expression to get the target value. In this case, for the joins and the lookups, I was stuck. So I dropped that. Can this be worked around?
Context variable question is answered. I can use context variables in the jobscript as well. It was kind of a silly question to ask.
Thanks in advance!
You can use Cache or Hash Component to load you rules and use them in combination with tJava or tJavaRow to apply the rules on the individual rows of data by splitting them based on delimiters in columns.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Pick up some tips and tricks with Context Variables
Take a look at this video about Talend Integration with Databricks
Learn how<SPAN>to modernize your Cloud Platform for Big Data Analytics with Talend and Microsoft Azure</SPAN>