This article explains how using a new feature in Talend 6.3 that makes the SDLC process simpler when building Big Data Jobs.
Contexts groups were available for Big Data Metadata in earlier versions, but the values would not get replaced when the context group was changed. This has been resolved in Talend 6.3. The process to centralize Hadoop metadata by making use of context variables and groups is shown below.
Set up Kerberos and get a ticket:
Get a ticket:
For details about these steps, see these instructions: How to use Kerberos in Talend Studio with Big Data v6.x.
At the bottom of the Cluster Connection screen, click Export as context.
On the Create/Reuse a context group screen, select Create a new repository context, then click Next
On the next screen, give the context group a name. Click Next, then Finish.
Double-click the context group to open it. Now you can start adding the cluster information such as NameNodeUri and ResourceManager, as shown below:
The Configure Contexts window opens. Click New at the bottom, give the context a name, then click OK.
Now you can add values for the new cluster environment:
If you want to know how Talend has bundled the libraries necessary for both context groups, open the Hadoop connection in Metadata, then click the ellipses [...] to the right of Use custom Hadoop configurations.
There will be one library for each context group.
Build a Standard Job as shown below. Add the tHDFSConnection and tHDFSOutput big data components and connect them.
For the tHDFSConnection component, select Repository as the Property Type, and select the metadata connection that you created in the process above. As you will see, all the context variables will be added automatically to the configuration.
On the Run tab of Studio, select which cluster you want the Job to run on by selecting the context group from the drop-down list. These are the contexts groups you created while configuring Hadoop.
Similarly, you can select the context while creating an Execution task or an Artifact task in TAC, as shown below: