I have to process N number of files 2 in parallel and 10 can be in queue

Four Stars BN
Four Stars

I have to process N number of files 2 in parallel and 10 can be in queue

I have to process N number of files from directory and I have to do  2 files processing in parallel. How do I make sure that every time different files are picked up from directory for processing, when I process these files in parallel.

Seven Stars

Re: I have to process N number of files 2 in parallel and 10 can be in queue

by Debug Run-->Traces Run.

Regards
Aashish
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Forteen Stars

Re: I have to process N number of files 2 in parallel and 10 can be in queue

@BN,in tFileList, you can specify filemask and get the n. number of files to iterate the next process by taking the global map current filename from tFileList.

Manohar B
Don't forget to give kudos/accept the solution when a replay is helpful.
Nine Stars

Re: I have to process N number of files 2 in parallel and 10 can be in queue

If you use tFileList and iterate through each file, on the iterate connection between the tFileList and the next component, click the "Enable parallel execution" and set the number to 2.

Regards David
Dont forget to give Kudos when an answer is helpful or mark the answer as the solution.
Four Stars BN
Four Stars

Re: I have to process N number of files 2 in parallel and 10 can be in queue

Thanks David,

   For some reason my "Enable parallel execution" is greyed out. Have to check why.

 

Nine Stars

Re: I have to process N number of files 2 in parallel and 10 can be in queue

Maybe check the job parameters tab, Extra, Multi thread execution.

Regards David
Dont forget to give Kudos when an answer is helpful or mark the answer as the solution.
Four Stars

Re: I have to process N number of files 2 in parallel and 10 can be in queue

Do we have any solution to this problem? I also want to process N files at the same time. So if the 1st file processing by the 1st subjob the second file should be processed by 2nd subjob and so on.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Modern Data Engineering in the Cloud

Learn about modern data engineering in the Cloud

Watch Now

How to deploy Talend Jobs as Docker images to Amazon, Azure and Google Cloud reg...

Learn how to deploy Talend Jobs as Docker images to Amazon, Azure and Google Cloud registries

Blog

Talend API Services Publish to Talend Cloud

Learn how to publish your API Services to Talend Cloud

Watch Now