Iterate a job multiple times with different input for each run

Six Stars

Iterate a job multiple times with different input for each run

Hi all,

 

I have a job that makes a download of folders and sub folders based on a variable that contains the current date.

 

Now, based on the current date, I want to download the same structure for each day based on a range of time.

 

E.g.: today is 2018/09/24 and I'm interested on the folders from today up to a week ago.

1) I start from today so I download all the folders (containing files) for 2018/09/24

2) yesterday was 2018/09/23 so I'll download all the related folders (and subfolders)

3) then the same as above but for the day before and so on until a week

 

I have the single job, how can I iterate it with a range of time?

 

I started evaluating the tForEach, but I'm not sure that is the right direction.

 

Any suggestion is appreciated.

 

Thanks,

Nick

Highlighted
Seven Stars

Re: Iterate a job multiple times with different input for each run

I would use a tJavaFlex.
Six Stars

Re: Iterate a job multiple times with different input for each run

hi navds, thank you for your hint. But my question would be: why and how to apply it? 

 

This is the job flow I have in place with a short description:

unarchive_job.PNG

 

 

I want to execute it n times, where n is equal to the number of days contained within a range of time. According to the example above, it would be 7 days. But in general it could be a month, or 2 weeks...

Seven Stars

Re: Iterate a job multiple times with different input for each run

  • Why ? Because it is simple
  • How ? Like this:

Screenshot from 2018-09-25 11-59-10.png

You just need to replace the tLogRow with tRunJob where you pass the date as context.

Cloud Free Trial

Try Talend Cloud free for 30 days.

Tutorial

Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.