Iterating through files

One Star

Iterating through files

I have a job right now that loads three CSV files into a DB and then run a series of jobs against that to build out a data warehouse. In the future there may be as many as 5 or 10.
Right now I have a separate job running for each CSV file. What I would like is a single job that goes through a folder running on each CSV that it finds. Does someone have a sample on this?
Thanks,
Luke
One Star

Re: Iterating through files

The component you want is tFileList. In that component you can specify a file mask and hook the iterate output to the rest of your processing.
One Star

Re: Iterating through files

Hi, does anyone know if there is a way I can pass the output from tFileList into a tRunJob component?
One Star

Re: Iterating through files

Ah, looks like you can add a context variable in the job, then set it as a parameter to the tRunJob and then pass this through an iteration between the tFileList and tRunJob:
((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))
One Star

Re: Iterating through files

I've set up a cron job that pulls AppStore Sales reports to a folder,
which I then let a reoccurring Talend job:
looking in, pulling out files, extracting, loading into a staging table, moves the used files.

-- Update --
Hmm, can't see the files I uploaded...

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog