Processing mutliple csv files through tS3Get

Five Stars

Processing mutliple csv files through tS3Get

DELETED


Accepted Solutions
Employee

Re: Processing mutliple csv files through tS3Get

Hi,

 

    You cannot go directly from tS3Get to tMap. tS3Get is used to bring the files from S3 to local for further processing and its job is transportation of file from one area to another.

 

    So you will need a tFileInputDelimited component to read the file once its available in local area to be processed further using tMap.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

 

View solution in original post


All Replies
Employee

Re: Processing mutliple csv files through tS3Get

Hi,

 

    You will have to use tS3List and please refer the example below.

image.png

 

image.png

 

image.png

 

In the tS3Get, you will have to specify the current key from previous component (use Control+Space to see the list). In this way, you can move multiple files in iterative fashion.

 

On a different note, I have seen that you have given On Component Ok to process the downstream flow. I would recommend to use On SubJob OK (from the starting of previous subjob's first component) so that you can allow the entire SubJob to be complete before moving to next activity.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Five Stars

Re: Processing mutliple csv files through tS3Get

Thank you Nikhil.  This helps a ton.  I'll give this a shot.  I basically just inherited all of our Talend Jobs that were controlling our BI data flow and was told "here it is, have fun, good luck..."  So thank you for baring with me on what now seems to be pretty fundamental.

Five Stars

Re: Processing mutliple csv files through tS3Get

Also, I see a ((String)globalMap.get(*component name*)) being used a lot.  I take it this is Java code and it's effectively taking the results of that component and formatting them in a string name to be passed into whatever parameters you are using for your components down stream?

Employee

Re: Processing mutliple csv files through tS3Get

Hi,

 

     No worries at all. Talend community is quite vibrant and you don't be alone in this journey :-)

 

     Please also spare a second to mark the topic as resolved if you are happy with the solution.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Five Stars

Re: Processing mutliple csv files through tS3Get

Is it possible for me to actually go right from the tS3Get to tMap or must there be a physical source that tMap needs to be able to read from?  Basically I'm asking is going from tS3Get to tFileInputDelimited the appropriate way to proceed with what I'm trying to accomplish?

Employee

Re: Processing mutliple csv files through tS3Get

Hi,

 

    You cannot go directly from tS3Get to tMap. tS3Get is used to bring the files from S3 to local for further processing and its job is transportation of file from one area to another.

 

    So you will need a tFileInputDelimited component to read the file once its available in local area to be processed further using tMap.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

 

View solution in original post

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog