[resolved] Parallel execution using mssql query in talend

Highlighted
One Star

[resolved] Parallel execution using mssql query in talend

Hi
I am using talend open studio and wanted to execute query in parallel threads which is getting use in tmssqlinput. I am using ms sql server.
How can i use parallelism or execute/run query in threads.
Regards
Vipin

Accepted Solutions
Seventeen Stars

Re: [resolved] Parallel execution using mssql query in talend

There are two major methods:
1. Cluster the input data sets by timestamps or IDs and start multiple (parallel) jobs for those closed value ranges.
2. Do not cluster the data sets and use the new feature of Talend called parallelisation (only available in the enterprise edition).
I prefer the first method because this way you can clearly estimate how long it will take to get everything and if one data cluster fails all others are probably not affected. The failed one could be repeated.
For method one it is a good design to have a kind of plan-table where you insert datasets describing the input data set cluster. The actual transfer jobs read the plan table and start with the processing for one of its entry.

All Replies
Seventeen Stars

Re: [resolved] Parallel execution using mssql query in talend

There are two major methods:
1. Cluster the input data sets by timestamps or IDs and start multiple (parallel) jobs for those closed value ranges.
2. Do not cluster the data sets and use the new feature of Talend called parallelisation (only available in the enterprise edition).
I prefer the first method because this way you can clearly estimate how long it will take to get everything and if one data cluster fails all others are probably not affected. The failed one could be repeated.
For method one it is a good design to have a kind of plan-table where you insert datasets describing the input data set cluster. The actual transfer jobs read the plan table and start with the processing for one of its entry.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Why Companies Move to the Cloud: 7 Success Stories

Learn how and why companies are moving to the Cloud

Read Now