One Star

Strategy for load balancing

which best practice / strategy does Talend propose for load balancing?
We have DI jobs that have to process big amounts of data and I would like to distribute the load on multiple execution servers. I.e. the same job on several Jobservers with an intelligent balancer in front deciding which server / job instance to use.
Is the Service Locator i.e. a distribution over the endpoint of an upstream webservice the only possibility to implement this?
(Greatly possible I only would like to know ... ;-) )
Many thanks
Community Manager

Re: Strategy for load balancing

Hi Rabe
Talend does not distribute a job and make it to run multiple job server, a job instance is always executed in the selected job Server, in Talend, the load balancing mechanism is implemented on a module called Grid Conductor (is available in Talend Administrator Center), it optimizes the scalability and availability of the integration processes by ensuring an optimal use of the execution grid.
The grid conductor relies on the definition of virtual servers, which group available resources, regardless of the system type (CPU, OS...).
Via a constant monitoring of the resources available on the execution servers, Grid Conductor guarantees that all jobs execute smoothly at triggering time and fully leverage available resources, removing bottlenecks created by the traditional single-server approach.
This alleviates any concerns related to resource preemption when a large number of jobs run concurrently, or when non-dedicated servers are used. Grid Conductor also provides automatic fail-over in the event an execution resource becomes unavailable.

Talend | Data Agility for Modern Business
One Star

Re: Strategy for load balancing

Hi Shong,
perfect! This is exactly the answer I was looking for / I needed. Thank you!
One final little question: the Grid Conductor is not a documented or visible feature in the TAC but an implicit concept, (or internal implementation module) linked with virtual servers, right? Or are there real views / dialogs dealing with a Grid Conductor (maybe invisible with my license) or documents complementing the TAC Guide? In my installation + documentation there is no physical Grid Conductor ...
Anyway your answer helped a lot!
Best regards