I'm wondering what options I have for kicking off threads in Talend for jobs? Currently we are using TIS 3.0, but will be upgrading to professional edition 3.1 soon. After playing around with other options, I starting to think that Talend is not the right tool and perhpas I'll do threading on my own and call the talend job externally. What I want is a thread pool and to apply a variable number of jobs to those threads. This is very much like Queues and Message Driven Beans in Java. It would be nice to quickly configure the thread pool up or down to optimize for a specific hardware. I've seen the tParallelize, but that seems to apply when you have a known number of parallel paths. What I have is a single job that can is going to be run in each thread (x-times) and the parameters passed into the job. I attached some image examples of what I want where I iterate for each file found and spawn a thread to process each file in its own thread. This however appears to be un-supported in talend based on other posts since components are synchronized. I also see the MOM Input, but not sure that threading is configurable. Here is the output for my simple example where it finds 7 files, but as you see the jobs are processed serially as explained in other posts. starting child, sleeping 10 seconds: Thread-12 finishing child: Thread-12 starting child, sleeping 10 seconds: Thread-14 finishing child: Thread-14 starting child, sleeping 10 seconds: Thread-16 finishing child: Thread-16 starting child, sleeping 10 seconds: Thread-18 finishing child: Thread-18 starting child, sleeping 10 seconds: Thread-20 finishing child: Thread-20 starting child, sleeping 10 seconds: Thread-22 finishing child: Thread-22 starting child, sleeping 10 seconds: Thread-24 finishing child: Thread-2
Re: What options do I have for multi threading jobs
My simple example that I posted does in fact work if you have the correct version of TIS. We were using TIS v 3.0.2 and it appears there was a bug that prevented mult-threading both tRunJob and tParallelize. Once I upgraded to TIS 3.1 Team Edition, my simple example began to multi-thread and in fact the thread pool is working as desired. We are seeing some concurrency problems in our sub jobs and we'll have to look at fixing those.