[resolved] Parallel execution using mssql query in talend

Highlighted
One Star

[resolved] Parallel execution using mssql query in talend

Hi
I am using talend open studio and wanted to execute query in parallel threads which is getting use in tmssqlinput. I am using ms sql server.
How can i use parallelism or execute/run query in threads.
Regards
Vipin

Accepted Solutions
Seventeen Stars

Re: [resolved] Parallel execution using mssql query in talend

There are two major methods:
1. Cluster the input data sets by timestamps or IDs and start multiple (parallel) jobs for those closed value ranges.
2. Do not cluster the data sets and use the new feature of Talend called parallelisation (only available in the enterprise edition).
I prefer the first method because this way you can clearly estimate how long it will take to get everything and if one data cluster fails all others are probably not affected. The failed one could be repeated.
For method one it is a good design to have a kind of plan-table where you insert datasets describing the input data set cluster. The actual transfer jobs read the plan table and start with the processing for one of its entry.

View solution in original post


All Replies
Seventeen Stars

Re: [resolved] Parallel execution using mssql query in talend

There are two major methods:
1. Cluster the input data sets by timestamps or IDs and start multiple (parallel) jobs for those closed value ranges.
2. Do not cluster the data sets and use the new feature of Talend called parallelisation (only available in the enterprise edition).
I prefer the first method because this way you can clearly estimate how long it will take to get everything and if one data cluster fails all others are probably not affected. The failed one could be repeated.
For method one it is a good design to have a kind of plan-table where you insert datasets describing the input data set cluster. The actual transfer jobs read the plan table and start with the processing for one of its entry.

View solution in original post

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog