How to access data in batches

One Star

How to access data in batches

First, I am new to component creation and I have very specific requirement. I am creating new component where I need to send data to web services. I am able to do so but I am sending data as single row in each request and this is not acceptable solution. My plan is to read multiple rows, send it to web-services for processing, retrieve the results, and then send it to output. I am not sure, how I can do this. Same time, web-services can be bottle neck and I can read data in multiple batches and spawn multiple threads to send request to web-services from each thread. So it it possible. Can I use any existing component for reference. Any help would be appreciated. Thanks
For ex, I have a .csv file having 10000 rows and I need to processing row no 2 using my web-server. Ideally, what I want is read data in 1000?s batch, create 5 threads to make web-services call and send/retrieve data, and put it in output in same order that it received.
Moderator

Re: How to access data in batches

Hi,
Sorry for the delay.
There is no such a component just for "thread process" but usually we set the option ?Enable parallel execution' on iterate to achieve that.
The job flow is tfileinputdelmited--main--tFlowToiterate--iterate--tFixedFlowInput--main--tWebservice--tLogRow
check out the option 'Enable parallel execution' on iterate link.
See the following screenshot
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.