Using Azure Functions to trigger Talend Cloud Jobs

Overview

This article explains how to use Azure Functions to trigger a Job managed by the Talend Cloud platform. It explains how to use a trigger on Azure Blob to execute a Job published on Talend Cloud. The Job is designed in Talend Studio and published to the Talend Cloud platform where its execution on the cloud or a remote engine is managed.

 

It covers:

  • Publishing a Job to Talend Cloud
  • Creating an Azure function based on a Blob trigger using the JavaScript language
  • Testing the integration end-to-end

 

Architecture

The diagram shows the technical implementation architecture of the solution described in this article.

arch.jpg

 

Prerequisites

Microsoft Azure

  • You need access to Microsoft Azure with a valid account (portal.azure.com).
  • You need full access to Azure services.

Talend

 

Create a Job with Talend Studio

This section explains how to create a new Job and publish it to Talend Cloud.

 

Configure Talend Studio to connect to Talend Cloud

In Studio:

  1. Navigate to Window > Preferences > Talend > Integration Cloud.
  2. Fill in your:

    • Account Username
    • Account Password

      Creating a job on Talend Studio 1.jpg

  3. Select the Advanced check box. This will display the Service URL drop-down list:

     integCloud.png

    Choose the correct Data Center for your location:

    • US Data Center
    • EMEA Data Center
    • <Custom>, add the correct URL for your environment

 

Create and publish a Job to Talend Cloud

This Job shows the logic of reading the file from Azure Storage, applying a transformation, and saving the file back to Azure Storage.

 

Since you retrieve the file locally onto the cloud or a remote engine where the Job is executed, any files from your previous runs must be removed. Thus, the Job starts by deleting any temp files from previous runs, retrieving and then deleting the file from Azure Storage, transforming it locally on the cloud or remote engine, and then uploading it into Azure Storage again. Your Job should look like this:

 

Publishing a Job to Talend Cloud 1.jpg

 

When you are ready to publish your Job, right-click the Job in the Repository and click Publish to Cloud.

Publishing a Job to Talend Cloud 2.jpg

 

If you have published the Job before, you may need to increment the version number of the Job before clicking the Finish button, as shown below:

large.jfif

 

Manage your deployment

Navigate to the login page for Talend Cloud: integrationcloud.talend.com. If you prefer to use the European instance, click the Go to European instance link.

 

  1. Type your Email and Password, then click the LOGIN button to connect.

    Publishing a Job to Talend Cloud 4-1.jpg

     

  2. Click the Launch link to access Talend Cloud.

    Publishing a Job to Talend Cloud 5.jpg

     

  3. Verify that your Job was published successfully.

 

Design an Azure Function App

This section explains how to create an Azure Function App that will call or trigger the Job when a new file arrives in Azure Storage.

 

Create the Azure Function App

  1. Open your web browser and type: portal.azure.com

    Create an Azure Function App 1.jpg

     

  2. Click Create a resource.

    Create an Azure Function App 2.jpg

     

  3. Configure your Azure Function App settings as follows:

    Field

    Value

    App name

    Define the name of your Azure Function App

    Subscription

    Select your Subscription

    Resource Group

    Select your Resource Group

    OS

    Select Windows

    Hosting Plan

    Select Consumption Plan

    Location

    Select your location

    Storage

    Select Azure Storage

    Application Insights Location

    Select your location

     

  4. Click Create.

    Create an Azure Function App 4.jpg

     

Write the Azure Function Logic

This section explains to write the logic that your Azure Function will run when it is executed.

 

  1. Navigate to your Azure Function App and click + (Create new).

    Create an Azure Function App 4.jpg

     

  2. On the Blob trigger pane, select JavaScript.

    Create an Azure Function App 5.jpg

     

  3. Configure your Function settings as follows:

    Field

    Value

    Language

    Leave JavaScript selected

    Name

    Define the name of your Function

    Path

    Define the name your Blob as:

    talend-in/{name}

    Storage account connection

    Select your Azure Storage

     

    Create an Azure Function App 6.jpg

     

  4. Click Create.
  5. Click your Function.

    Create an Azure Function App 7.jpg

     

  6. Replace the default code with the following, remembering to change the username and password and changing the body.executable parameter to match your specific configuration:

    function callback(error, response, body) {
    context.log(JSON.stringify(body));
    context.log(response);
    }
    
    module.exports = function (context, myBlob) {
    context.log("JavaScript blob trigger function processed blob \n Name:", context.bindingData.name, "\n Blob Size:", myBlob.length, "Bytes");
    
    // Body
    var parameters = new Object();
    parameters.fileNameIn = context.bindingData.name;
    var body = new Object();
    body.executable = "5b23c0d3b1187f0ebec41ee1";
    body.parameters = parameters;
    
    var jsonString= JSON.stringify(body);
    context.log(jsonString);
     
    // Call POST
    
    // Include the request library for Node.js
    var request = require('request');
    // Basic Authentication credentials
    var username = "*****@talend.com";
    var password = "*****";
    var authenticationHeader = "Basic " + new Buffer(username + ":" + password).toString("base64");
    
    //Request
    var options ={
    method: 'POST',
    url : "https://ipaas.us.cloud.talend.com/api/v1.1/executions",
    body: jsonString,
    headers : {
    "Content-Type": "application/json",
    "Accept": "application/json",
    "Authorization" : authenticationHeader }
    };
    
    request(options, callback);
    context.log("Done!")
    
    context.done();
    };

     

    Create an Azure Function App 7-1.jpg

     

 

Create the package.json file

Create and place the following package.json file in the shared folder of your Azure Function.

{
     "name": "sample-cloud-storage",
     "version": "0.0.1",
     "dependencies": {
          "request": "2.87.0"
      }
}

The screen below shows the package.json file as it should be configured and located:

Create an Azure Function App 8.jpg

 

Configure the Platform Feature

  1. Navigate to your Azure Function and click Platform features.

    Create an Azure Function App 9.jpg

     

  2. In the Platform features section, click Console.

    Create an Azure Function App 10.jpg

     

  3. Type dir in the console, then press Enter.

    Create an Azure Function App 11.jpg

     

  4. Confirm that the package.json file is in the directory.

    Create an Azure Function App 12.jpg

     

  5. Type npm install request, then press Enter.

    Create an Azure Function App 13.jpg

     

You should see the output in the console as shown in the screenshot; if not, consult your Azure documentation and retrace your steps to make sure the function is properly configured. Talend recommends reading through the Azure Functions documentation, as configuration steps may change in the future.

 

End-to-end processing of the file

The integration you just developed is event-driven: as soon as a file is placed in Azure Storage, it will trigger the Azure Function, which will trigger the Job. Review the logs to see the execution of these event tasks. You should get an output file uploaded into Azure Storage.

 

INPUT

Place a file in the Blob Container (talend-in) as shown below:

Test 1.jpg

 

Check the Azure Function Log

To confirm that the Azure Function has executed, you need to check its log.

Test 2.jpg

 

Review Talend Job Execution

Click the Log link, to the right of the execution record, and confirm the Job executed successfully. If any errors are encountered, they are shown in the Log.

Test 3.jpg

 

The Log window opens and displays the Job results, as shown below:

Test 4.jpg

 

OUTPUT

Go to the Blob container (talend-out) and confirm that the output file was written by the Talend Job as expected.

Test 5.jpg

 

Conclusion

This article explains how to build an end-to-end integration based on triggers with Azure Functions and a Talend Job, and shows a way to perform serverless computing. It is transparent for the IT team, and there is no need to maintain and host a server instance. Note that Talend Cloud provides Cloud Engines which are fully managed by Talend, and also provides Remote Engines if you want to host your own execution engine. However, this is not needed, unless you have security or other constraints about the data being processed that require you to host your own.

 

Microsoft Azure and Talend Cloud platforms make it easy for you to perform serverless trigger and event-based processing.

Version history
Revision #:
13 of 13
Last update:
‎09-28-2018 11:58 PM
Updated by:
 
Labels (2)