Deadlock during multiple Talend job execution

Not applicable

Deadlock during multiple Talend job execution

I have talend jobs configured through Talend Admin Console (TAC) in multiple instances through context groups like UK, SA etc. For these instances I have set the file triggers (like source file trigger, confirmation file trigger etc). For Eg, UK instance is dependent on both UK source file and a confirmation file from a dependent process say X. 
But these 2 instances can run in parallel (since the UK and SA source files can come in parallel). When these instances am trying to insert/update into the same SQL server 2012 database table at the same time, one of the instance is failing with the deadlock error as below: 
“java.sql.SQLException:Transaction (Process ID 69) was deadlocked on lock | communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction.” .
Hence I need to manually run the failed instance again in such cases.
Hence I am looking out for an option on TAC side itself if TAC can handle such type of parallel runs/deadlocks and wait for the other process/instance to complete successfully before proceeding with the current process/instance (so that I do not get this deadlock error and hence avoid this manual re-run).
Also, I need to know if TAC has a “RETRY” job option – which basically re-runs the failed jobs automatically “N” number of times  (where “N” can be configured) – This property is present in few other schedulers and hence need to check if TAC has this kind of property.
Any inputs which will help us to resolve this issue is appreciated.
Thanks in advance.
One Star

Re: Deadlock during multiple Talend job execution

You can commit your SQL output for each and every row.(using Advanced Settings).That would prevent tables to wait in loop. But this might degrade your performance!

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey


Talend named a Leader.

Get your copy


Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables


How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration


Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.