Submit Multiple Big Data batch job on Azure Databricks cluster

Highlighted
Four Stars

Submit Multiple Big Data batch job on Azure Databricks cluster

Hi ,

 

We have requirement of submitting multiple spark big data batch jobs on single Azure Databricks cluster. We tried submitting more than one job at the same  time to single databricks cluster, but jobs start failing with the issue :

“org.talend.bigdata.launcher.utils.BigDataLauncherException: Status 400 for /api/2.0/clusters/restart”

 

After some analysis I figured out that as per the talend official documentation, we can’t run multiple big data batch jobs in parallel on single databricks cluster. Please find the link below:

https://help.talend.com/reader/iLJ_d4Eb9ZgJEEFVKwPNig/uG6WXACe~Vz1fYBBPjpnPA

 

Is there a way I can run big data jobs in parallel on a single databricks cluster?

 

 

Moderator

Re: Submit Multiple Big Data batch job on Azure Databricks cluster

Hello,

Could you please create a new feature jira issue on talend bug tracker?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Talend Cloud Developer Series – Deploying First Job to Cloud

This video will show you how to run a job in Studio and then publish that job to Talend Cloud

Watch Now

Talend Cloud Developer Series – Fetching Studio License

This video will help someone new to using Talend Studio get started by connecting to Talend Cloud and fetching the Studio License

Watch Now

Talend Cloud Developer Series - Introduction

The Talend Cloud Developer Series was created to give you a solid foundational understanding of Talend’s Cloud Integration Platform

Watch Now