One Star

[resolved] Parallel execution using mssql query in talend

Hi
I am using talend open studio and wanted to execute query in parallel threads which is getting use in tmssqlinput. I am using ms sql server.
How can i use parallelism or execute/run query in threads.
Regards
Vipin
1 REPLY
Seventeen Stars

Re: [resolved] Parallel execution using mssql query in talend

There are two major methods:
1. Cluster the input data sets by timestamps or IDs and start multiple (parallel) jobs for those closed value ranges.
2. Do not cluster the data sets and use the new feature of Talend called parallelisation (only available in the enterprise edition).
I prefer the first method because this way you can clearly estimate how long it will take to get everything and if one data cluster fails all others are probably not affected. The failed one could be repeated.
For method one it is a good design to have a kind of plan-table where you insert datasets describing the input data set cluster. The actual transfer jobs read the plan table and start with the processing for one of its entry.