Submit Multiple Big Data batch job on Azure Databricks cluster

Four Stars

Submit Multiple Big Data batch job on Azure Databricks cluster

Hi ,

 

We have requirement of submitting multiple spark big data batch jobs on single Azure Databricks cluster. We tried submitting more than one job at the same  time to single databricks cluster, but jobs start failing with the issue :

“org.talend.bigdata.launcher.utils.BigDataLauncherException: Status 400 for /api/2.0/clusters/restart”

 

After some analysis I figured out that as per the talend official documentation, we can’t run multiple big data batch jobs in parallel on single databricks cluster. Please find the link below:

https://help.talend.com/reader/iLJ_d4Eb9ZgJEEFVKwPNig/uG6WXACe~Vz1fYBBPjpnPA

 

Is there a way I can run big data jobs in parallel on a single databricks cluster?

 

 

Highlighted
Moderator

Re: Submit Multiple Big Data batch job on Azure Databricks cluster

Hello,

Could you please create a new feature jira issue on talend bug tracker?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Talend Cloud Developer Series – Defining Metadata

This video focuses on different methods of adding metadata to a job in Talend Cloud

Watch Now

Talend Cloud Developer Series – Updating Context Variables

This video will show you how to add context parameters to a job in Talend Cloud

Watch Now

Talend Cloud Developer Series – Deploying First Job to Cloud

This video will show you how to run a job in Studio and then publish that job to Talend Cloud

Watch Now